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A Message trom the President 


AAU institutions are distinguished for their world-class research and the qual- 
ity of their graduate education programs. Less recognized and valued is the 
role AAU universities play in undergraduate education. Yet AAU insti- 
tutions educate close to 1.2 million undergraduate students every aca- 

demic year. In educating these undergraduate students, AAU universi- 

ties have a responsibility to promote the use of evidence-based teach- 

ing practices proven by research to be most effective at advancing stu- 

dent success. Additionally, they must provide their faculty members with 

11% of undergraduates the encouragement, training, and support to effectively employ these 
AAU universities educate. (Structional approaches in the classroom. The AAU Undergraduate 
1.2 million STEM Education Initiative is a significant test of the degree to which a 
Undergraduates group of prominent research universities can work collectively with their 
national organization to improve the quality of teaching in undergraduate STEM 

courses, especially large introductory and gateway courses, thereby enhancing 

the learning experiences of many thousands of their undergraduate students. 


The results of the Initiative thus far indicate a resoundingly affirmative answer 
to this test. At the same time, the Initiative has helped AAU understand how 
it, aS a Major association of research universities, can help to support meaning- 
ful change at various institutional levels to improve undergraduate STEM edu- 
cation. While there is much work to be done to realize a ‘new normal’ — one 
characterized by personal and institutional expectations that all faculty mem- 
bers will both use and be rewarded for using evidence-based approaches to 
instruction — our Initiative suggests that progress is being made. 


AAU continues its commitment to achieving widespread systemic change in 
this area and to promoting excellence in undergraduate education at major 
research universities. We are now reaching a major tipping point. We cannot 
condone poor teaching of introductory STEM courses because we are trying 
to weed out the weaker students in the class or simply because a professor, 
department and/or institution fails to recognize and accept that there are, in 
fact, more effective ways to teach. Failing to implement evidence-based teach- 
ing practices in the classroom must be viewed as irresponsible, an abrogation 
of fulfilling our collective mission to ensure that all students who are interested 
in learning and enrolled in a STEM course — not just those who will choose to 
major in or pursue an advanced degree in that discipline — are provided with 
the maximum opportunity to succeed. This change is what is needed to ensure 
that we have the STEM-literate workforce and general population required to 
propel the nation forward into the 21% century and beyond. 


Sue perme 


Mary Sue Coleman 
President 
Association of American Universities 
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Objective 


To intluence the culture of 
STEM departments at AAU 
Institutions so that faculty 


members are encouraged 

and supported to use teaching 
practices proven by research 
to be effective in engaging 
Students in STEM education 
elalemiamal=)ie)iaremcidere(-lalecm(<\-aa) 
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Develop an effective analytical framework for 
assessing and improving the quality of STEM 
iu=t-(e@allatem-larem(=t-leallale pm er-1au@0l-la\mlamealom iat 
two years of college. 


Support AAU STEM project sites at a subset of 
AAU universities to implement the Framework, 
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universities committed to implementing 

STEM teaching and learning reforms. 
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can use to train, recognize, and reward faculty 
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effectiveness of their STEM teaching. 


Work with federal research agencies to develop 
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efforts to improve undergraduate learning. 


Develop effective means for sharing information 
about promising and effective undergraduate 

Sy =i =re LU (er hd (olay e)aeole|e-laakyar-] ©) e)cey- (eal eam anl-adalelery 
Tare eX=tel-le tele l (=e 


Undergraduate STEM Education Initiative 


Executive Summary 


In September 2011, the Association of American 
Universities launched a major initiative to improve 
undergraduate STEM education. The overall 
objective was to influence the culture of STEM 
departments at AAU institutions so that faculty 
members are encouraged and supported to 

use teaching practices proven by research to 

be effective in engaging students in STEM 
education and in helping students learn. 


The Undergraduate STEM Education Initiative’s intent was to help research 
universities better assess and improve the quality of teaching in STEM fields 
by: promoting the use of teaching techniques in STEM classes demonstrated 
by scholarship to be the most effective at engaging and helping students 
learn; encouraging universities and STEM departments to better evaluate, 
recognize and reward faculty members for the quality and effectiveness of 
their teaching; and facilitate the creation of an effective network for dissem- 
inating and sharing best practices in undergraduate STEM education reform 
and classroom based educational improvements. 


AAU's Initiative was not launched with the intention of producing yet another 
report. Instead, its aim was to encourage and support research universi- 
ties to act upon and implement recommendations already made in national 
reports.’ 2 The initiative was, in some sense, an experiment to see if as a 
leading association of research universities, AAU could facilitate meaningful 
and long-lasting systemic change in undergraduate STEM education by pro- 
viding its members with a framework accompanied by additional tools, sup- 
port, and encouragement. 


To help with this effort, AAU established a project team and convened an 
advisory committee composed of experts in undergraduate STEM teaching 
and learning. It also established five overarching goals for the initiative: 

. Develop an effective analytical framework for assessing and improving the 
quality of STEM teaching and learning, particularly in the first two years 
of college. 


= 


= 


President’s Council of Advisors on Science and Technology (PCAST), Engage to Excel: Producing 

One Million Additional College Graduates with Degrees in Science, Technology, Engineering, and 
Mathematics (Washington, DC: PCAST, 2012). 

2 National Research Council (NRC), Discipline-Based Education Research: Understanding and Improving 
Learning in Undergraduate Science and Engineering (Washington, DC: National Academies Press, 2012). 
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2. Support AAU STEM project sites at a subset of AAU universities to imple- 
ment the Framework, and develop a broader network of AAU universi- 
ties committed to implementing STEM teaching and learning reforms. 

3. Explore means that institutions and departments can use to train, recog- 
nize, and reward faculty members who want to improve the quality and 
effectiveness of their STEM teaching. 

4.Work with federal research agencies to develop means of recognizing, 

rewarding and promoting efforts to improve undergraduate learning. 

. Develop effective means for sharing information about promising and 

effective undergraduate STEM education programs, approaches, meth- 
ods, and pedagogies. 


uo 


Since the Initiative was launched, AAU has received 11 grants ($7.9M) from 
private and corporate foundations and the federal government to advance the 
Initiative. Progress Toward Achieving Systemic Change provides a five-year sta- 
tus report on the AAU Undergraduate STEM Education Initiative. 


Engagement by AAU Universities 
AAU universities have demonstrated widespread enthusiasm and interest in 
the Initiative. 


Participation in the Initiative by AAU member institutions has been high. This 
is demonstrated by the fact that all 62 AAU institutions have designated a 
STEM Campus Point of Contact to serve as a liaison between AAU and his 
or her campus for the Initiative. 


To date, 55 AAU member universities have participated in the Initiative, 
including more than 450 unique faculty members and institutional leaders. 


42 institutions convened campus stakeholders (e.g., faculty members and 
administrators) to respond to our request for comments on the initial 
Framework draft. 


Over half of AAU’s membership is active in the AAU STEM Network and 
more than 275 unique faculty members and institutional leaders have par- 
ticipated in network events. 


31 institutions submitted proposals to be considered as a STEM Project Site 
and to receive STEM Network Mini-grants. 


AAU universities are engaged in multiple innovative efforts to improve under- 
graduate education and to help all students succeed. For example, 


= department-wide innovations to undergraduate STEM courses are becom- 
ing institutional priorities; 


= teaching and learning centers are being redesigned to better support 
department educational reform efforts; 
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® data infrastructures and analytics are being capitalized on to improve stu- 
dent learning; 


= campuses are exploring new hiring practices to advance improvements in 
STEM education; 


= learning spaces are being developed and re-engineered; and 


= campuses are addressing the critical challenge of improving the evaluation 
of faculty teaching. 


AAU STEM Project Sites 
Central to the project site reforms was the role of the academic department 
and its faculty members. 


Seed-funding was awarded to eight AAU member project sites (Brown 
University; Michigan State University; The University of Arizona; University of 
California, Davis; University of Colorado Boulder; University of North Carolina 
at Chapel Hill; University of Pennsylvania; and Washington University in St. 
Louis) to implement reforms that address the core elements of the Framework 
and specific challenges facing their campus in undergraduate STEM education. 


= Over three academic years the eight STEM Project Sites have engaged 39 
departments, reformed 162 courses, involved 230 faculty members and 
1,676 learning assistants (graduate and undergraduate), and transformed 
STEM courses for 138,531 student-seats. 


= Project sites reported trends toward improved learning gains, decreased 
failure rates, improved persistence from introductory to later courses, and 
narrowing achievement gaps especially for women, under-represented 
minorities, and first-generation college students. 


# All project sites increased the number of courses targeted for reform 
based on evidence-based pedagogy, and increased the number of fac- 
ulty members (tenure-track and non-tenure track) participating. One-half 
of project sites expanded their reach to additional departments which 
were not originally included in their proposals. One-half of project sites 
developed and disseminated common tools used to assess teaching and 
instruction. Additionally, several project sites linked co-curricular activities 
with reformed courses to increase retention in STEM majors. 


Across the eight project sites, use of graduate and undergraduate assis- 
tants in active learning classes more than doubled, from 740 to 1,676, 
during the three years of the AAU project. Inclusion of undergraduate and 
graduate students in instructional roles has benefits for institutions at the 
level of the course or section. With more trained individuals in the room, 
the capacity to facilitate and evaluate evidence-based pedagogy increases. 
The experience also benefits the students themselves by reinforcing core 
concepts and helping them to learn effective teaching practices. 
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== | Examining the Impact 
= | Evaluation is a key component of the Initiative. 
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AAU is assisting member universities in tracking the progress of their reform 
efforts in addition to evaluating the overall impact of the STEM Initiative. 


To document cross-institutional effects, AAU collected data from all project 
sites over a three-and-a-half-year period. Common data collection included 
a survey of instructor attitudes and practices in participating departments; 
department chair narratives on policy and practice to assess teaching in the 
promotion and tenure process; and campus and department level assess- 
ment of learning spaces. Findings are presented on pages 60-61, 82-83, 89. 


AAU collected annual reports and conducted two site visits at each of 
the eight project sites to allow for a more qualitative evaluation of project 
implementation and progress. In total, AAU met and talked with 325 indi- 
viduals across the eight project sites. Information from site visits, common 
data collection, and project site reports are among the sources used to 
provide much of the detailed analysis found in Section 2. 


In partnership with Adrianna Kezar, Professor, Rossier School of Education 
and Co-Director, Pullias Center for Higher Education at the University of 
Southern California, AAU has examined the role that a national higher edu- 
cation association can play in promoting and scaling systemic institutional 
reforms in undergraduate teaching and learning. A summary of the project 
is on pages 124-125. 


Progress Toward Institutional Change 
AAU universities are successfully implementing strategies to achieve long-lasting 
improvements in undergraduate STEM teaching and learning. 


Based upon a comprehensive review of the undergraduate STEM education 
reforms occurring at AAU universities, we found the following key elements 
at various institutional levels to be important factors in improving the quality 
and effectiveness of undergraduate STEM teaching and learning. 


Shift from individual to collective responsibility by departments for 
introductory course curriculum. AAU has observed that departments 
most likely to emphasize evidence-based active-learning strategies in foun- 
dational courses have thought deeply about the curricula and content of 
these courses, along with ways to assess student learning. Ultimately, col- 
lective responsibility for shared learning objectives by course will necessitate 
developing a uniform vision of educational improvement among faculty 
members within and across departments, as well as the development of 
mechanisms to assess progress in teaching effectiveness for all students. 


Hire educational experts within departments to bolster reforms. One 
strategy to successfully institutionalize reform is to embed instructional exper- 
tise within the department to provide educational leadership and to support 
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all faculty members in the adoption and use of evidence-based pedagogy. 
Although the types of appointments of individuals with this type of expertise 
vary widely (e.g., tenure-track, non-tenure track, junior and senior ranks), 
these individuals all have in common an understanding of the discipline and 
how students learn best within the discipline. When used most effectively, 
these individuals are well positioned to provide educational leadership to 
the department. 


Harness institution-wide data to support student learning. Research 
universities can greatly facilitate STEM education improvement by supporting 
the development and use of institution-wide data and analytical tools on stu- 
dent instruction and learning outcomes. Keys to successful use of data analytics 
include: 1) distinguishing between the types of data useful for individual faculty 
members designing and assessing their courses and the types of data used in 
departmental decision-making; 2) the ease and efficiency of use are essential 
to broad acceptance of teaching-related metrics; and 3) data should not be 
seen as sufficient in their own right but must be used to help make decisions 
and establish policies aimed at advancing educational improvement. 


Reorganize administrative support services to better support depart- 
mental reform efforts. Another key to successful institutionalization of 
undergraduate instructional reforms is to align relevant administrative units, 
such as Centers for Teaching and Learning, with department-based instruc 
tional improvement efforts. Providing college or campus-wide structures to 
support departmental reform efforts increased the likelihood of institution- 
alization in AAU project sites. 


Develop and re-engineer learning spaces. Creating collaborative learn- 
ing spaces to support evidence-based pedagogy has been a catalyst for fac 
ulty members to reflect on how they teach their courses. Students have also 
attributed learning gains to classroom environments that foster engagement 
and interaction. 


Better manage the simultaneous pursuit of high quality teaching and 
research. The development and use of more effective ways to evaluate teach- 
ing quality and effectiveness in the faculty reward structures will be required 
to institutionalize STEM educational reforms. 


Commit to systemic and long-term STEM reforms. Institutionalizing 
reforms of undergraduate STEM education at research universities eventu- 
ally requires internal institutional investment and alignment of resources; it 
cannot be achieved solely by a series of isolated externally-funded grants. 
Public pronouncements of support for these undergraduate reforms by 
university leaders also contributes to the spread of instructional reforms 
across departments. Without further institutional commitment, however, 
such pronouncements fail in achieving meaningful and long lasting reform. 
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= Leverage AAU to advance educational reforms and institutional change. 
AAU involvement has symbolic implications that can help campuses achieve 
cultural and institutional change by providing legitimacy to STEM education 
reform efforts. 


Resources for Universities 
AAU has developed resources to help research universities take a systems 
perspective to improving undergraduate STEM education. 


To help facilitate institutional change, key resources and tools have been 
developed. These include: 


= The Framework for Systemic Change in Undergraduate STEM Teaching 
and Learning provides a change model for improving the quality and 
effectiveness of STEM teaching and learning at research universities.? The 
Framework recognizes the wider setting in which educational innovations 
take place — the department, the college, the university and the external 
environment — and addresses key institutional elements necessary for 
sustained improvement to undergraduate STEM education. 


= Fssential Questions and Data Sources for Continuous Improvement of 
Undergraduate STEM Teaching and Learning helps member campuses 
track the progress of their reform efforts.* This resource complements 
the Framework and provides a set of key questions designed to engage 
institutional leaders and faculty members in discussions about teaching 
and learning. The report also provides data sources and analytical tools 
available to answer these questions and inform decision-making, as well as 
provides guidance to address common challenges in evaluating the quality 
and effectiveness of undergraduate education. 


® Aligning Practice to Policies provides specific guidance to departments and 
institutions on how to implement new methods for evaluating, recognizing, 
and rewarding teaching at research universities, particularly relating to how 
teaching is judged for purposes of promotion, tenure, and annual reviews.° 
This resource was developed in collaboration with the Cottrell Scholars 
funded by Research Corporation for Science Advancement. 


3 Association of American Universities (AAU), Framework for Systemic Change in Undergraduate STEM 
Teaching and Learning (Washington, DC: AAU, 2013). 

4 Association of American Universities (AAU), Essential Questions and Data Sources for Continuous 
Improvement of Undergraduate STEM Teaching and Learning (Washington, DC: AAU, 2017). 

5 Association of American Universities (AAU), Aligning Practice to Policies: Changing the Culture to 
Recognize and Reward Teaching at Research Universities (Washington, DC: AAU, 2017). 
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=S | Sustaining the Momentum 
AAU is committed to improving STEM education at research universities. 
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AAU leadership has committed to extend the initial five-year undergraduate 
STEM effort indefinitely by integrating continued support for undergraduate 
STEM education reform and improvement into its ongoing staffing structure 
and portfolio of work. AAU will also look to broaden its efforts to improve 
undergraduate instruction beyond STEM fields in the future. 


A major award provided an additional 24 AAU universities with institutional 
mini-grants to further advance and coordinate existing efforts aimed at 
improving undergraduate STEM teaching and learning. 


AAU actively engages a broader network of faculty members and admin- 
istrators at AAU universities committed to improving undergraduate STEM 
teaching and learning. The AAU Undergraduate STEM Education Network 
has convened conferences annually to discuss innovative practices to improve 
STEM education and hosted a variety of targeted workshops to address 
critical issues. A workshop, convened by AAU, brought together STEM 
department chairs from AAU universities to share information about and 
discuss improving STEM teaching within their departments and recognizing 
and rewarding faculty members for the quality and effectiveness of their 
teaching. Moving forward AAU will convene the AAU Undergraduate STEM 
Education Network and STEM Department Chairs on alternating years. 


AAU will continue to collaborate with other national associations, organi- 
zations, funders, and industry partners to coordinate activities relating to 
undergraduate STEM reform and to develop effective means to disseminate 
promising and effective programs, approaches, methods, and strategies. 
The Initiative is engaging multiple stakeholders to promote long-lasting 
reform to undergraduate STEM education and working to address the cul- 
tural and policy barriers within research universities that hamper educational 
improvement and innovation. 


In partnership with federal agencies, AAU is finding new ways to engage 
faculty members to broaden the impact of their research by becoming more 
innovative in the classroom, teaching more effectively, and providing for 
authentic research experiences to undergraduate students. 


An ongoing examination of how universities can successfully coordinate 
multiple undergraduate STEM education reforms to achieve sustainable 
change is underway. This project is designed in recognition of the reality 
that many AAU universities are advancing multiple department-level as 
well as institution-wide efforts to improve undergraduate STEM teaching 
and learning. 


Association of American Universities 


Report Road Map 
section 1 


Section 1 provides information about the approach and process undertaken 
by AAU to develop, launch, and implement an initiative in collaboration with 
our member universities to improve the quality and effectiveness of under- 
graduate teaching and learning in science, technology, engineering, and 
mathematics (STEM) fields. 


section 2 


Section 2 provides detailed description and analysis of the eight seed-funded 
AAU STEM project sites. Included in this section are cross-cutting strate- 
gies aligned to core elements of AAU's Framework for Systemic Change in 
Undergraduate STEM Teaching and Learning and a number of vignettes profil- 
ing specific institutional approaches and supporting evidence of their impact. 


Section 3 


Section 3 discusses the broader AAU STEM Network and highlights additional 
institutional commitments to advancing the goals of the Initiative. 


Section 4 examines both evidence of and keys to successful institutionaliza- 
tion of undergraduate STEM education reforms at two levels — the campus 
and AAU as an association. 


Online Appendix 


The online appendix contains numerous documents used to implement the 
Initiative. Throughout the four sections of the report specific appendix mate- 
rials are referenced. www.aau.edu/STEM/online-appendix 
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Timeline 
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Informal survey 

Ol VNO MUTANS 1ES116 
undergraduate STEM 
retention programs. 


Framework 


AAU STEM 
Project Sites 
& Network 


2011 


June 
Hunter R. Rawlings, III 
becomes AAU President. 


July 

Foundational work is 
carried out to define 

the AAU Undergraduate 
STEM Education Initiative. 


A matrix of undergraduate 

STEM reform efforts is de- 

veloped, a discussion draft 
disseminated, and an advi- 
sory committee formed. 


September 
The AAU STEM Initiative 
is publicly announced. 


October 
Advisory committee 
conference call. 


November 

AAU starts to develop a 
Framework to guide the 
Initiative. 


2012 


February 
Advisory committee 
in-person meeting. 


April 

AAU asks member Presi- 
dents/Chancellors to ap- 

point a campus liaison for 
the AAU STEM Initiative. 


May 

Coalition for Reform of 
Undergraduate STEM 
Education convenes higher 
education associations 
engaged in improving 
undergraduate STEM 
education. 


November 

Emily Miller joins AAU 
as Project Manager for 
AAU STEM Initiative. 


October 

AAU requests feedback 
by member universities on 
draft Framework. 


A) | 


March 
Advisory committee 
in-person meeting. 


September 
Advisory committee 
conference call. 


January 

AAU’s Framework for 
Systemic Change in 
Undergraduate Teaching 
and Learning is finalized. 


February 

Call for AAU STEM 
Initiative Project Site con- 
cept paper submissions. 


June 
Eight project sites are 
awarded and announced. 


July 
AAU STEM Network 
conference. 


Development of cross- 
instutional measures 

and metrics to support 
the multi-instutional 
evaluation of project sites. 


September 

First round of campus 
visits to project sites 
commence. 


Feedback sought on 
common data measures 
and metrics from project 
site teams. 
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2014 


March 
Advisory committee 
conference call. 


July 
aau.edu/stem 
website launched. 


September 

AAU in collaboration with 
Adrianna Kezar, University 
of Southern California Pro- 
fessor of Higher Education 
began to examine the role 
of a national association in 
scaling institutional reforms 
to STEM education. 


February 

Request for common 
data from project sites— 
Time Point 1. 


May 
Workshop for project site 
teams held at AAU. 


July 
AAU STEM Network 
conference. 


2015 


August 
Advisory committee 
in-person meeting. 


April 
Flex travel grants offered 
to project sites. 


AAU hosts a workshop 
for AAU STEM department 
chairs. 


September 

Second round of visits 

to project sites by the 
AAU STEM Initiative team 
commence. 


October 

AAU STEM Network 
conference co-hosted 
with WashU. 


2016 


June 
Mary Sue Coleman 
becomes AAU President. 


July 

No-cost extensions 
issued to all eight project 
sites for 4th year. 


September 

Request for common 
data from project sites— 
Time Point 2. 


Z O | Lae 


May 
Advisory committee 
in-person meeting. 


June 

Essential Questions 

and Data Sources for 
Continuous Improvement 
of Undergraduate STEM 
Teaching and Learning 

is released by AAU. 


January 

AAU STEM Network 
mini-grant projects on 
12 campuses awarded. 


March 
AAU receives final reports 
from project sites. 


October 
AAU STEM Network 
conference. 
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Timeline 
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Dissemination 


al elarel are 
Sources 


2011 


2011-Present 

Over 50 presentations to 
disciplinary societies, fed- 
eral agencies, Congress, 
higher education institu- 
tions and associations, and 
national coalitions and net- 
works working to improve 
undergraduate education. 


2012 


May 

The National Academies’ 
practitioner guide to the 
DBER report profiles the 
AAU Undergraduate STEM 
Education Initiative. 


May 

Sponsored by Research 
Corporation for Scientific 
Advancement (RCSA), 
AAU hosts a meeting for 
funders to learn more 
about the STEM Initiative, 
$15K. 


October 

The Leona M. and 
Harry B. Helmsley 
Charitable Trust, 
$5M for 3 years. 


Zo) | 


May 

National Science 
Foundation WIDER, 
$295K for 2.5 years. 


August 
RCSA Cottrell Scholars 
Collaborative, $25K for 
2 years. 


July 

The Burroughs 
Wellcome Fund, 
$50K for 1 year. 
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2014 


January 

AAU and RCSA 
Cottrell Scholars 
host a workshop 
on the evaluation 
of teaching. 


September 
Presentation at AAU 
Provost meeting about 
institution-wide data and 
analytics. 


October 

Coalition for Reform of 
Undergraduate STEM 
Education releases 
Sourcebook on Achieving 
Systemic Change. 


December 

The New York Times 
publishes an article about 
the AAU Undergraduate 
STEM Education Initiative. 


September 2014 
National Science 
Foundation IUSE, 
$600K for 3 years. 


2015 


July 

AAU & RCSA publish 
Searching for Better 
Approaches: Effective 
Evaluation of Teaching 
and Learning in STEM. 


AAU & RCSA Cottrell 
Scholars publish comment 
article in Nature. 


August 

Inside Higher Ed publishes 
an article about the AAU 
Undergraduate STEM 
Education Initiative. 


October 

Chapters focusing on 

the work of the AAU 
Undergraduate STEM 
Education Initiative are 
published in Transforming 
Institutions: Undergraduate 
STEM Education for the 
21st Century. 


April 

AAU Department Chair 
workshop sponsored 
by Elsevier, $10K. 


August 

RCSA Cottrell Scholars 
Collaborative I, 

$25K for 2 years. 


2016 


May 

AAU and RCSA Cottrell 
Scholars host a workshop 
on aligning policy and 
practice to promote effec- 
tive evaluation of teaching. 


October 

AAU responds to the White 
House Office of Science and 
Technology Policy’s Active 
Learning Call to Action. 


December 

American Innovation and 
Competitiveness Act ad- 
dition of “and instruction” 
updates NSF’s broader 
impacts criteria. 


January 

The Leona M. and 
Harry B. Helmsley 
Charitable Trust, 
$240K for 1.5 years. 


November 
Northrop Grumman 
Foundation, $1M for 
4 years. 


October 

National Science 
Foundation IUSE, 
$700K for 3 years. 


Zz O) | riven 


February 

AAU & RCSA publish 
Improving Undergradu- 
ate STEM Education at 
Research Universities: A 


Collection of Case Studies. 


July 

AAU & RCSA Cottrell 
Scholars publish 
Aligning Practice to 
Policies: Changing the 
Culture to Recognize 
and Reward Teaching 

at Research Universities. 


September 

AAU staff publish Cata- 
lyzing Institutional Trans- 
formation: Insights from 
the AAU STEM Initiative in 
Change: The Magazine of 
Higher Learning. 
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AAU Frames 
tne Problem 


The impetus for AAU to undertake a national 
initiative to improve the quality and effective- 
ness of undergraduate STEM education at 
research universities was several-fold. 


It was grounded in an increasing national focus on STEM education combined 
with a set of high level policy reports calling for improvements in undergrad- 
uate STEM education, a growing body of scholarship on teaching and learn- 
ing in the STEM fields, and concern that research universities were particularly 
vulnerable to public criticism about the quality and effectiveness of under- 
graduate STEM teaching. Of particular concern were the high attrition rates 
of undergraduate students at major research universities who would declare 
STEM majors but who would subsequently drop out of STEM fields or fail to 
complete a degree in any field, with many of them attributing their decisions 
in part to the poor quality of faculty instruction. 


At the time AAU initiated its efforts, many students who intended to major 
in a STEM field were not completing their degrees, or completing degrees in 
non-STEM disciplines. According to National Science Foundation (NSF) data, 
university enrollments continue to increase, as do numbers of bachelor’s 
degrees awarded in both STEM and non-STEM fields. However, STEM degrees 
as a proportion of total bachelor’s degrees have remained relatively constant 
at about 15-17 percent. Moreover, the proportion of freshmen intending to 
major in STEM fields exhibits a similar pattern, remaining relatively constant 
at around 25 percent over the past 15 years. This gap between the percent- 
age of freshmen who intend to major in STEM fields and the percentage of 
awarded bachelor’s degrees in those fields is a persistent trend. 
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In the 2005 Survey of the American Freshman, as reported by the House 
Science Committee, half of all students who began in the physical or biolog- 
ical sciences and 60 percent of those in mathematics dropped out of these 
fields by their senior year, compared with a 30 percent drop-out rate in the 
humanities and social sciences.® According to Talking About Leaving: Why 
Undergraduates Leave the Sciences, by Elaine Seymour and Nancy M. Hewitt, 
44 percent of entering freshmen in 1987 who intended to major in a STEM 
field switched to a non-STEM major by 1991 (this percentage varies somewhat 
among specific STEM disciplines); for non-STEM majors, only about 30 percent 
switched to another group of majors.’ The Higher Education Research Institute 
(HERI) reported that only 38 percent of students who entered STEM bache- 
lor’s programs in the 1993-1994 academic year earned a bachelor’s degree 
in a STEM field within six years. The HERI analysis also showed that, across all 
races, students who started in STEM fields were less likely to complete degrees 
in any field than students who intended to major in non-STEM fields.® 


AAU reflected on the question: Why do so many students who enter college 
intending to major in a STEM discipline fail to earn a bachelor’s degree in STEM? 
As reported by the Information Technology and Innovation Foundation (ITIF), 
several studies have shown that most students who leave STEM do so between 
the first and second year, rather than later in their college career. Seymour and 
Hewitt surveyed students and obtained the now-infamous result that 90 per- 
cent of students who switched out of STEM fields cited poor teaching as a 
concern. ITIF summarizes Seymour and Hewitt’s results: “Of the 23 most com- 
monly cited reasons for switching out of STEM, all but 7 had something to do 
with the pedagogical experience.” Undergraduate teaching was clearly a major 
factor in students choosing to leave STEM fields, and because most students 
who leave STEM do so during the first two years of college, those years are 
especially critical in terms of teaching. 


This pattern continues to be borne out by more recent data and reports. 
According to a National Center for Education Statistics (NCES) study that exam- 
ined attrition rates in STEM majors for students who began college in the 2003- 
2004 academic year, 48% of students who entered a bachelor’s degree pro- 
gram in STEM between 2003 and 2008 had left STEM fields by spring 2009.° 
Roughly half of these leavers switched their major to a non-STEM field, and 
the others exited college before earning a degree. 


The National Research Council appointed the Committee on Barriers and 
Opportunities in Completing 2-Year and 4-Year STEM Degrees to address the 
barriers that prevent students from earning the STEM degrees to which they 
aspire and to identify opportunities to promote completion of undergraduate 


a 


US House of Representatives Committee on Science and Technology, Hearing Charter: Strengthening 

Undergraduate and Graduate STEM Education (Washington, DC, February 4, 2010). 

7 Elaine Seymour and Nancy M. Hewitt, Talking about Leaving: Why Undergraduates Leave the Sciences 
(Boulder, CO: Westview Press, 1997). 

8 Higher Education Research Institute (HERI), Degrees of Success: Bachelor’s Degree Completion Rates 

among Initial STEM Majors (Los Angeles, CA: HERI, 2010). 

National Center for Education Statistics (NCES), STEM Attrition: College Students’ Paths Into and Out of 

STEM Fields (Washington, DC: NCES, 2013). 
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STEM degrees.’ The committee concluded that there is an opportunity to 
expand and diversify the nation’s STEM workforce and STEM-skilled work- 
ers in all fields if there is a commitment to appropriately support the diverse, 
complex pathways students take to earn STEM degrees. 


At the same time, AAU staff had long recognized that its member institu- 
tions were vulnerable to criticism on undergraduate STEM teaching, learning, 
and retention such as those raised in the 1998 Boyer Commission Report on 
educating undergraduates in the research university." Also, the ever-growing 
national discourse to justify the cost and value of an undergraduate degree 
at a research university was a topic of discussion among the AAU leader- 
ship. STEM fields are critical to generating the ideas, products, and industries 
that drive our nation’s global competitiveness, and with the passage of time, 
they are becoming even more crucial to our country’s success. Therefore, it is 
important that students, who will comprise our future workforce and leaders, 
are educated using the best and most effective methods in STEM education. 
Universities must encourage students who enter college intending to major in 
a STEM field in their educational pursuits, and support the fundamental STEM 
literacy of students pursuing non-STEM majors. Moreover, schools must work 


to broaden participation in STEM ; é 
fields of study. Institutions havea A comprehensive meta-analysis 


responsibility to ensure that any of of 225 studies revealed ‘hat 


their students can learn in STEM 


classrooms and pursue careers in underg raduate students in 

SIEM TISICS IT MEV CESIETO GOSS: else acai irecuonal eciuikes 
The latest research on teach- are 1.5 times more likely to fail 
ing ante "eating mas aso len ahan students inc asses that 


the development of instructional 
methods that are more engag- Use active learning methods. 
ing and effective at helping stu- 

dents learn. This effect has been extensively documented in STEM fields.’ 

13 A comprehensive meta-analysis of 225 studies revealed that undergrad- 
uate students in classes with traditional lectures are 1.5 times more likely 

to fail than students in classes that use active learning methods."* Also, 

a growing body of evidence demonstrates that learning gains from using 
these teaching approaches in highly structured classrooms are particularly 
good for students from disadvantaged and diverse backgrounds and that 


10 National Academies of Sciences, Engineering, and Medicine (NASEM), Barriers and Opportunities 

or 2-Year and 4-Year STEM Degrees: Systemic Change to Support Students’ Diverse Pathways 

Washington, DC: National Academies Press, 2016). 

Boyer Commission on Educating Undergraduates in the Research University, Reinventing Undergraduate 

Education: A Blueprint for America’s Research Universities (Stony Brook, NY, 1998). 

12 National Research Council (NRC), Discipline-Based Education Research: Understanding and Improving 

Learning in Undergraduate Science and Engineering (Washington, DC: National Academies Press, 2012). 

13 President's Council of Advisors on Science and Technology (PCAST), Engage to Excel: Producing 

One Million Additional College Graduates with Degrees in Science, Technology, Engineering, and 
athematics (Washington, DC: PCAST, 2012). 

14 Scott Freeman, et al., Active Learning Increases Student Performance in Science, Engineering, and 
athematics, Proceedings of the National Academy of Sciences of the United States of America 111 

2014): 8410-8415. 
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active learning confers disproportionate benefits for female students in male- 
dominated fields.'>"8 


Furthermore, the national policy environment has begun to reflect a more 
coordinated effort to improve undergraduate STEM education across relevant 
organizations and actors.'?-*4 There has been a shift away from isolated direc 
tives within individual disciplines and nationally funded efforts that do not 
require long-lasting reforms within academic institutions. Today many funders 
are designing solicitations with expectations for projects to build and sustain 
institutional change.?° 


As AAU reflected on STEM undergraduate education in the 2009-2012 time- 
frame, it found that despite the problem of students leaving STEM fields and 
the movement toward developing and supporting systemic reform in STEM 
undergraduate education to address growing public pressures, a majority 
of university STEM faculty members who teach undergraduate science and 
engineering classes remained inattentive to the shifting landscape. Student- 
centered, evidence-based teaching practices were not yet the norm in most 
undergraduate STEM education courses, and the desired magnitude of change 
in STEM pedagogy had not materialized.**?° 


A principal reason for the lack of widespread pedagogical reform in STEM is 
the use of theoretical perspectives whose focus is primarily on individual fac 
ulty members and the students in their classrooms.?° Much of this literature 


15 Sarah L. Eddy, and Kelly A. Hogan, Getting Under the Hood: How and For Whom Does Increasing Course 
Structure Work? CBE-Life Sciences Education 13 (2014): 453-468. 

16 David C. Haak, Janneke HilleRisLambers, Emille Pitre, and Scott Freeman, Increased Structure and Active 

Learning Reduce the Achievement Gap in Introductory Biology, Science 332 (2011): 1213-1216. 

17 Kyle F. Trenshaw, David M. Targan, and James M. Valles, Closing the Achievement Gap in STEM: A Two- 

Year Reform Effort at Brown University, Proceedings of the American Society for Engineering Education 

Northeast Section Conference (2016): 1-9. 

18 Mercedes Lorenzo, Catherine H. Crouch, and Eric Mazur, Reducing the Gender Gap in the Physics 

Classroom, American Journal of Physics 74 (2006): 118-122. 

19 American Association for the Advancement of Sciences (AAAS), Vision and Change in Undergraduate 

Biology Education: A Call to Action (Washington, DC: AAAS, 2009). 

20 Jodi L. Wesemann, and Mary M. Kirchoff, Chemistry Education: Transforming the Human Elements 

Washington, DC: Association of American Colleges & Universities, 2011). 

21 American Society of Engineering Education (ASEE), Transforming Undergraduate Education in 
Engineering: Phase 1: Synthesizing and Integrating Industry Perspectives (Arlington, VA: ASEE, 2013). 

22 National Research Council (NRC), A New Biology for the 21% Century (Washington, DC: National 
Academies Press, 2009). 

23 National Research Council (NRC), The Mathematical Sciences in 2025 (Washington, DC: National 

Academies Press, 2013). 

24 National Science and Technology Council (NSTC), Federal Science, Technology, Engineering and 

athematics (STEM) Education: 5-Year Strategic Plan (Washington, DC: NSTC, 2013). 

25 Catherine L. Fry, Achieving Systemic Change: A Sourcebook for Advancing and Funding Undergraduate 

STEM Education (Washington, DC: The Coalition for Reform of Undergraduate Education, 2014). 

26 Winston A. Anderson, et al., Changing the Culture of Science Education at Research Intensive 

Universities, Science 331 (2011): 152-152. 

27 National Research Council (NRC), Discipline-Based Education Research: Understanding and Improving 

Learning in Undergraduate Science and Engineering (Washington, DC: National Academies Press, 2012). 

28 Melissa Dancy and Charles Henderson, Pedagogical Practices and Instructional Change of Physics 

Faculty, American Journal of Physics 78 (2010): 1056-1063. 

29 Melissa Dancy, Charles Henderson, and Julian Smith, Understanding Educational Transformation: Findings 

rom a Survey of Past Participants of the Physics and Astronomy New Faculty Workshop, Proceedings of 

the Physics Education Research Conference (2013): 113-116. 

30 Melissa Dancy and Charles Henderson, Beyond the Individual Instructor: Systemic Constraints in 

he Implementation of Research-Informed Practices, Proceedings of the Physics Education Research 

Conference (2004): 1-4. 
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Kelly Hogan, STEM Teaching Associate Professor and Assistant Dean of Instructional Innovation, teaching ina 
high-structure, high-engagement introductory biology classroom at the University of North Carolina at Chapel Hill. 


centers on micro-level assessments of the classroom, which is crucial to 
assessing the effect of pedagogy on student learning. Yet this literature often 
ignores the larger institutional and external environment and fails to account 
for the costs and political challenges in scaling up reforms." 72 Concern 
about more macro-level environments requires a change in assessment from 
looking solely for benefits and learning outcomes at the course or program 
level to a more nuanced consideration of factors that facilitate, impede, or 
influence wide-spread transformation in undergraduate STEM education. 


To increase the implementation and widespread adoption of instructional strat- 
egies shown to be effective requires a model of change that includes the roles 
of research evidence, leadership, resources, faculty workload and rewards, 
and faculty professional development. In this context, empirical evidence is 
only one part of the reform effort. As Fairweather has explained, “research 
evidence of instructional effectiveness is a necessary but not sufficient condi- 
tion” for faculty to change their teaching practices. Fairweather suggests that 
the assumption that “the instructional role can be addressed independently 
from other aspects of the faculty position, particularly research, and from the 


31 P. David Fisher, James S. Fairweather, and Marilyn J. Amey, Systemic Reform in Undergraduate 
Engineering Education: The Role of Collective Responsibility, International Journal of Engineering 
Education 19 (2003): 768-776. 

32 Charles Henderson and Melissa Dancy, Increasing the Impact and Diffusion of STEM Education 
Innovations, white paper commissioned for the Characterizing the Impact and Diffusion of Engineering 
Innovations Forum (February 7-8, 2011). 
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larger institutional context” is misguided.*? Given the size and scale of higher 
education, changing individual faculty members or even isolated departments 
will have minimal impact. To achieve long-lasting and broadly disseminated 
educational reforms, efforts must go well beyond this micro-level focus on 
faculty members. 


Scholars recommend that sustainable STEM education reform requires engag- 
ing institutional leaders such as department chairs, deans, and presidents in 
rethinking institutional structures and culture.*4 A recent case study of under- 
graduate STEM education reform conducted at the University of Colorado 
Boulder found that top-down (campus-level academic leaders) and bot- 
tom-up (faculty) reforms alone are 


inadequate for sustained institutional AAU'S approach of developing 
improvement in undergraduate edu- <ul 
cation; middle-out (chairs, college d shared priority aio 
deans) reforms are also required.» Multiple stakeholders rather 
Austin’ Il-d ted syst an ae 

vnach to chante alco Samect, tran only individual faculty 


approach to change also suggests 


that external stakeholders such as members offers a potentially 
disciplinary societies, government 


agencies, and employers are crucial transformative approach to 
to long-lasting change.** Insum, STEM education reform. 


transforming undergraduate STEM 

education requires multiple facilitators or “levers” pushing for change that 
can counterbalance the forces that sustain ineffective instructional practices 
and address the obstacles inherent in the system in which educational inno- 
vations take place.?7?9 


The AAU Undergraduate STEM Education Initiative launched in 2011 is spe- 
cifically aimed at assisting AAU institutions to implement what we already 
know works in STEM education, and assuring that these teaching prac 
tices are widely implemented in STEM departments to support the learning 
and persistence of students in STEM on a large scale. AAU’s approach of 


33 James Fairweather, Linking Evidence and Promising Practices in Science, Technology, Engineering, and 
Mathematics (STEM) Undergraduate Education, white paper commissioned for the National Academies 
National Research Council Board of Science Education Workshop: Evidence on Promising Practices in 
Undergraduate Science, Technology, Engineering, and Mathematics (STEM) Education (October 13-14, 2008). 

34 Ann E. Austin, Barriers to Change in Higher Education: Taking a Systems Approach to Transforming 
Undergraduate STEM Education, white paper commissioned for the Coalition for Reform of 
Undergraduate STEM Education Foundation Workshop on Transforming Undergraduate STEM Education 
Vune 17-19, 2013). 

35 Joel C. Corbo, Daniel L. Reinholz, Melissa H. Dancy, Stanley Deetz and Noah Finkelstein, Sustainable 
Change: A Model for Transforming Departmental Culture to Support STEM Education Innovation, 
arXiv:1412.3034 (December 9, 2014). 

36 Ann E. Austin. (2011). Promoting Evidence-Based Change in Undergraduate Science Education, paper 
commissioned for the Board on Science Education of the National Academies National Research Council 
(March 1, 2011). 

37 Winston A. Anderson, et al., Changing the Culture of Science Education at Research Intensive 
Universities, Science 331 (2011): 152-152. 

38 Andrea L. Beach, Charles Henderson and Noah Finkelstein, Facilitating Change in Undergraduate STEM 
Education, Change: The Magazine of Higher Learning 44 (2012): 52-59. 

39 Susan Elrod and Adrianna Kezar, Increasing Student Success in STEM: A Guide to Systemic Institutional 
Change (Washington, DC: Association of American Colleges & Universities, 2016). 
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developing a shared priority among multiple stakeholders rather than only 
individual faculty members offers a potentially transformative approach to 
STEM education reform. 


This ambitious project, which seeks to increase the importance of under- 
graduate STEM education in the nation’s top research universities, is pro- 
moting the implementation of a more systemic view of educational reform 
within academia.*° 


40 National Research Council (NRC), Designing Learning: A National Organization Leverages Systemic 
Change in STEM Teaching and Learning in Reaching Students: What Research Says about Effective 
Instruction in Undergraduate Science and Engineering (Washington, DC: National Academies Press, 
2015), 203-204. 
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AAU Takes 
a Systems 
Perspective 


The AAU STEM Initiative from the beginning has 
been informed by broader theoretical perspec- 
tives about organizational change in academia 
and about faculty work and rewards. 


AAU’'s Framework for Systemic Change to Undergraduate STEM Teaching 
and Learning recognizes the wider setting in which educational innovations 
take place — the department, the college, the university and the external 
environment — and addresses key institutional elements necessary for sus- 
tained improvement to undergraduate STEM education.*! The Framework 
was developed in collaboration with member universities. 


The core of AAU’s Framework is pedagogy: the practices used by faculty mem- 
bers to engage students and guide and support their learning. To successfully 
enact and institutionalize the use of evidence-based teaching techniques, two 
layers around this pedagogical core are necessary: scaffolding, or support, for 
both faculty and students, and larger cultural change to facilitate changing 
teaching practices. Ultimately, the Framework provides a set of key elements that 
need to be addressed to bring about sustainable change at an institutional level. 


41 Association of American Universities (AAU), Framework for Systematic Change in Undergraduate STEM 
Teaching and Learning (Washington, DC: AAU, 2013). 
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The Framework addresses organizational change at two levels—broadly, at 
the multi-institutional level and locally, at the university or college level. From 
a multi-institutional perspective, the Framework provides a commitment to a 
systems approach to change and a unifying goal, one that includes a shared 
understanding of the challenges and a set of key institutional elements that 
must be addressed to bring about sustainable change.*?-*4 Yet, the Framework 
allows for universities and colleges to use diverse strategies and approaches to 
achieve the common goal, as different strategies and approaches will be effective 
for achieving systemic improvement in STEM teaching and learning at different 
institutions. Thus, the Framework is highly adaptable and respects local insti- 
tutional culture and history. Visit aau.edu/STEM to see examples of innovative 
institutional efforts and how they are mapped to elements of the Framework. 


The Framework has proved central to several elements of the AAU Initiative, 
from guiding campus thinking to selection of the funded and highly visible 
project sites to recent efforts to organize information about resources available 
to campuses to facilitate continued conversation about change, especially 
surrounding the need for continual assessment of reform efforts. ll 


42 John Kania and Mark Kramer, Collective Impact, Stanford Social Innovation Review 9 (2011): 36-41. 
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GRANTS 


| AAU has received 11 grants totaling 
$7.9M from private foundations, corporate 
foundations, and the federal government 
mm tO advance tne AAU STEM Initiative. 


Tne Role 
ot AAU 


Foundational Work 


Hunter R. Rawlings Ill became the president of AAU in June 2011. At his 
first meeting with staff, he expressed a strong desire for AAU to undertake 
an initiative to improve STEM education at the undergraduate level at AAU 
universities. This desire meshed nicely with activities already underway and 
being actively pursued by Toby Smith, AAU’s Vice President for Policy under 
the leadership of Bob Berdahl, the previous AAU president. These activities 
included ongoing conversations with Carl Wieman, a Nobel Prize winning 
physicist with a passion for improving undergraduate STEM education, who 
had recently been appointed Associate Director for Science at the White House 
Office of Science and Technology Policy (OSTP); with Linda Slakey, Director 
of the Division of Undergraduate Education in the Education and Human 
Resources (EHR) Directorate of the National Science Foundation (NSF); and 
with the President’s Council of Advisors on Science and Technology (PCAST) 
who had created a working group and were in the midst of conducting their 
own report on the need to improve undergraduate STEM education. Moreover, 
Toby had previously conducted a survey of AAU member universities asking 
them to highlight noteworthy programs in undergraduate STEM education. 
Results of the survey indicated that most efforts occurred in out-of-classroom 
or co-curricular activities (e.g., undergraduate research opportunities, bridge 
programs, living-learning communities, etc.), which, although important, do 
not address the within-class experiences and the effectiveness of faculty 
members’ instructional practice and student engagement. 


During the summer of 2011, Josh Trapani, AAU’s Associate Vice President 
of Research and Policy Analysis, prepared a “discussion draft” in the 
form of a white paper of a proposal for a five-year initiative to improve 
undergraduate STEM Education.*° The discussion draft laid out the problem, 


45 Association of American Universities (AAU), Five-Year Initiative for Improving Undergraduate STEM 
Education: Discussion Draft (Washington, DC: AAU, October 14, 2011). 
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highlighting the fact that “improving undergraduate teaching is integral 
to meeting the pressing national need for more STEM majors.” The white 
paper pointed to previous work showing that many students interested in 
STEM switched to other majors during the first two years of college, and 
that teaching was one of the main causes for this shift. Cultural factors at 
research universities worked against instructors incorporating more active 
learning pedagogy in their classes, even though the evidence base supporting 
active learning pedagogy held appeal for researchers who were also teachers. 
The discussion draft laid out five goals, which have continued to guide the 
Initiative through its subsequent activity. 


From the beginning, AAU staff recognized that such an ambitious initiative 
would not succeed if it were simply an add-on to existing AAU activities. 
Staff lacked both expertise and bandwidth to perform the work, and the 
development of a demonstration program required an external funding 
source. Coincident with producing this discussion draft, AAU staff began 
to identify individuals who might serve on an advisory committee for the 
Initiative and help provide expert guidance. The initial membership of the 
advisory committee included practitioners and leaders in aspects of work 
related to the goals of the Initiative. The Initiative was announced publicly in 
September 2011, and the advisory committee held its first meeting, by phone, 
in October. Many of the individuals on the advisory committee provided 
critiques of the discussion draft. 


Kristen Hodge-Clark, a AAAS Fellow at AAU, gathered data to map major 
association and disciplinary society efforts in STEM reform and to identify 
areas of overlap among various organizations. As a part of this effort, AAU 
developed a matrix of STEM undergraduate education reform efforts at 
research institutions.*° 


Additionally, AAU drew on the expertise of federal officials and colleagues at 
other organizations, including associations, disciplinary societies, and funders, 
in honing the plan for the Initiative. In particular, staff coordinated with two 
advisory committee members, S. James Gates and Jo Handelsman, who at 
the time were also serving as co-chairs of the PCAST working group on issues 
around improving undergraduate STEM education. PCAST's report “Engage to 
Excel” was released in February 2012 and provided a national-level vision that 
was consistent with the goals of the AAU Initiative, especially in its focus on 
the importance of reforming teaching.*” The advisory committee held its first 
in-person meeting, also in February. In May, the National Academies released 
its report on discipline-based education research, which further supported the 
emphasis on active pedagogy in the Initiative.*® 


46 Association of American Universities (AAU), Matrix of National STEM Undergraduate Education 
Initiatives (Washington, DC: AAU, December 1, 2011). 

47 President's Council of Advisors on Science and Technology (PCAST), Engage to Excel: Producing 
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Mathematics (Washington, DC: PCAST, 2012). 
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AAU staff began compiling information that was envisioned as useful to 
developing a framework for institutions to assess and improve the quality of 
STEM teaching and learning. In placing framework development as the first goal, 
staff benefited from the example and experience of the Association of Public 
and Land-grant Universities’ (APLU) Science & Mathematics Teacher Imperative 
(SMTI).4° The framework was initially envisioned as a lengthy, detailed document 
and set of resources. The first rough outline of what would be included in a 
framework was produced in November, 2011. This version received feedback 
from members of the advisory committee and benefited from the information 
contained in the PCAST report. It was further revised during the spring and 
summer, and sent out to AAU member campuses for feedback in October, 
2012 [see Online Appendix for request for feedback on the Framework]. 


During this time-period, the AAU Initiative continued to gain momentum. In 
April 2012, AAU member university presidents and chancellors were asked 
at their spring meeting to provide a campus primary point of contact for 
the Initiative. This person was to be someone to whom the presidents and 
chancellors would rely on and engage with regarding the AAU Initiative. During 
that spring, AAU had conversations with numerous potential funders of the 
demonstration program. The Research Corporation for Science Advancement 
(RCSA) sponsored a workshop for funders interested in learning more about the 
AAU Initiative and other initiatives focused on improving undergraduate STEM 
education. This workshop was held at AAU in May, and attendees included The 
Leona M. and Harry B. Helmsley Charitable Trust, who were beginning to move 
into the space of higher education.*° Over the summer, AAU staff, along with 
two members of the advisory committee — Linda Slakey and James Fairweather, 
who served as a co-Pl — produced a proposal for The Helmsley Charitable Trust 
to conduct a demonstration project on AAU campuses. The same group also 
developed a proposal focused on metrics and evaluation for NSF. Ultimately, 
both proposals were successful. AAU announced the The Helmsley Charitable 
Trust grant in October, 2012, and the NSF WIDER grant in May, 2013. °° 


Implementation 


In fall of 2012, with The Helmsley Charitable Trust funding secured, AAU hired a 
full-time project manager for the Initiative. Emily Miller joined AAU in November. 
Collectively, Toby Smith, Jim Fairweather, Linda Slakey, Josh Trapani, and Emily 
Miller became the core AAU team responsible for the implementation of the 
AAU Undergraduate STEM Education Initiative and the corresponding goals. This 
group has met bi-weekly for a one-hour teleconference, and at least twice a 
year for a one or two day working session in Washington, DC, since fall of 2012. 


49 Association of Public and Land-Grant Universities (APLU), Developing the Analytic Framework: 
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DC: APLU, 2012). 

50 The Leona M. and Harry B. Helmsley Charitable Trust 
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The initial draft Framework was refined based on feedback provided by 42 
different AAU member campuses. In response to this feedback, the Framework 
was transformed from a detailed and proscriptive set of instructions to a 
concise conceptual document. The final Framework provides a set of key 
institutional elements that need to be addressed to bring about sustainable 
change. From a multi-institutional perspective, the Framework provides a 
shared vision for change, one that includes a common understanding of the 
challenges and an agreed upon set of institutional elements that must be 
addressed to bring about sustainable change. Along with the five goals in the 
discussion draft, the Framework became central to subsequent work. 


Collectively the project team developed and implemented a process for selecting 
project sites funded by The Helmsley Charitable Trust. From the beginning, AAU's 
approach to the demonstration project involved a balance: taking advantage of 
the competitive nature of AAU’s member universities with one another without 
creating a two-tiered system that might exclude some institutions to the extent 
of discouraging them from acting. The project sites were envisioned as one part 
of a larger STEM Network that included all interested AAU member universities. 
In February 2013, AAU put out a call for concept papers from schools interested 
in being part of the demonstration project [see Online Appendix for project 
site selection process materials]. Thirty-one AAU universities submitted concept 
papers, which were reviewed using a rubric and narrowed to 11 institutions, who 
were asked for more detailed plans of work. From these 11, eight project sites 
were selected in June 2013.% 


The eight project sites-Brown University; Michigan State University; The 
University of Arizona; University of California, Davis; University of Colorado 
Boulder; University of North Carolina at Chapel Hill; University of Pennsylvania; 
and Washington University in St. Louis-served as laboratories to implement 
the key elements of the Framework and represented the first phase of 
encouraging AAU universities to take a systems approach to reform of 
undergraduate teaching practices. Each of the eight project sites received 
$500,000 to seed fund projects (Year 1: $250K, Years 2-3: $125K). 


It is important to note that these eight universities were not selected because 
they were the most advanced in terms of activity already focused on improving 
undergraduate STEM education. AAU made a deliberate decision to create 
balance among the schools not only in terms of public/private status, region, size, 
and other factors, but also in terms of proposal objectives and how developed 
existing STEM teaching reform efforts on campus were. The hope was that 
the eight project sites could serve as potential models for other institutions, no 
matter where these others were in terms of emphasis and activity. 


53 AAU Selects Eight Campus Project Sites for Undergraduate STEM Education Initiative, AAU press 
release, June 25, 2013. 
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AAU's Initiative also recognized that large-scale improvements require a 
network through which ideas can travel, be tested, modified, and improved 
in a continuous cycle of growth. Research by Fairweather and by Eckel and 
Kezar strongly suggests that evidence alone is not sufficient for sustainable 
reform.°>°° Rather, peer relationships and institutional as well as interpersonal 
networks are crucial factors in changing ideas and practices.°” °* Thus, AAU 
also built into its approach the AAU STEM Network — a collaborative network 
that would allow AAU institutions that were not project sites to participate 
in the Initiative. It became clear as the process moved forward that several 
institutions — both project sites and non-project sites — were serious about 
advancing educational reforms and interested in learning from other AAU 
institutions that were tackling similar challenges on their own campuses. 


AAU has hosted three STEM Network conferences — in 2013, 2014, and 2015 
— with attendees including administrators, faculty members, postdocs, and 
students from AAU universities. These conferences have given attendees op- 
portunities to showcase their work and learn about the work of others, dis- 
cuss common themes and challenges, and build relationships across campus 
roles and institutions. Additionally, 


AAU frequently hosts workshops to A A\ J intentionally built a multi- 


provide in-person forums for all AAU : ; 
institutions to engage in the Initiative, instutional network dl med 


and AAU brings together on occasion at improving the quality and 


key stakeholders to address specific ; 

challenges to implementing institu- effectiveness of underg raduate 
tional efforts to reform STEM teach- STEM education in the nation’s 
ing and learning for undergraduate ie 

students. For example, AAU hosted a top research universities. 

two-day workshop in April 2015 for 

STEM department chairs and faculty members, “Improving Undergraduate STEM 

Teaching & Learning: The Role of the Department Chair.” This workshop, which 

was sponsored by Elsevier, provided over 100 department chairs, faculty mem- 

bers, and university administrators with an opportunity to discuss introductory 

curriculum redesign efforts, staffing models for introductory STEM courses, 

evaluation of department innovations in teaching and learning, and metrics for 

rewarding teaching. 


To date, 55 (out of 62) AAU member universities have participated in 
undergraduate reform activities hosted by AAU with involvement from 
more than 275 faculty members and institutional leaders. In addition, AAU 
recently awarded 12 mini-grants to universities who are engaged in the AAU 


55 James Fairweather, Linking Evidence and Promising Practices in Science, Technology, Engineering, and 
Mathematics (STEM) Undergraduate Education, white paper commissioned for the National Academies 
National Research Council Board of Science Education Workshop: Evidence on Promising Practices in 
Undergraduate Science, Technology, Engineering, and Mathematics (STEM) Education (October 13-14, 2008). 
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STEM Network but not part of the original eight project sites. AAU has 
learned through its STEM Initiative that grants awarded by AAU to member 
institutions can have powerful symbolic implications that can help campuses 
facilitate change. 


Toby Smith was invited to speak about the AAU Initiative during the 2012 
RCSA Cottrell Scholars conference. At the conference, Toby discussed with 
current and former Cottrell Scholars a major barrier to improving the quality of 
undergraduate education: the predominant use of student-based evaluations to 
assess teaching quality at colleges and universities. While effective at assessing 
faculty popularity, these student evaluations often fail to accurately reflect 
teaching quality and student learning. The two groups proposed a joint project, 
which RCSA subsequently agreed to support, aimed at identifying new and 
innovative means to evaluate and reward teaching quality and effectiveness. 


To help to accomplish the goals of this joint project, AAU and a subset of 
scholars from the RCSA Cottrell Scholars Collaborative held a joint workshop 
in January 2014 that brought together leading research-active faculty members 
as well as higher education scholars and practitioners to understand the 
landscape of established and emergent means to reward teaching more 
accurately than traditional measures such as student evaluations and to assess 
how research universities do or do not reward teaching in promotion and 
tenure decisions. The workshop culminated in a workshop report, Searching 
for Better Approaches: Effective Evaluation of Teaching and Learning in STEM 
and an article in Nature titled, University Learning: Improve Undergraduate 
Science Education.°*? © The Cottrell Scholars and AAU were awarded a second 
collaborative project by RCSA and in May 2016 brought together leading higher 
education scholars and practitioners as well as research-active faculty members 
to develop specific recommendations and guidance to value, assess, and reward 
effective teaching.®! 


In addition to AAU’s efforts to try to find ways to better assess and evaluate 
teaching, AAU has also worked to better leverage existing NSF broader impacts 
requirements to improve the quality of undergraduate STEM education and to 
achieve meaningful and long-lasting cultural change. 


Currently, all NSF grant proposals are evaluated on two broad-based 
criteria: their intellectual merit, which encompasses the potential to advance 
knowledge, and their broader impacts. The broader impacts criterion 
encompasses the potential of the work being done associated with the grant 
to benefit society and contribute to the achievement of specific, desired 
societal outcomes. 


59 Association of American Universities (AAU) and Research Corporation for Science Advancement 
(RCSA), Searching for Better Approaches: Effective Evaluation of Teaching and Learning in STEM 
(Tucson, AZ: RCSA, 2015). 

60 Stephen E. Bradforth, et al., University Learning: Improve Undergraduate Science Education, Nature 
523 (2015): 282-284. 
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As a part of 2010 legislation reauthorizing the America COMPETES Act, Congress 
included eight specific goals for NSF broader impacts criteria. Two of these goals, 
“improving undergraduate education” and “increasing public scientific literacy,” 
directly relate to the AAU Initiative, and can be used as a tool to encourage 
faculty to improve the quality of their undergraduate STEM teaching. 


There are, however, many challenges in getting faculty members to utilize 
improving their classroom teaching practices to fulfill their NSF broader 
impacts requirements. These challenges include the fact that many NSF 
researchers are unfamiliar with the broader impacts criteria and unaware 
that improving the quality of their classroom teaching, if done correctly, can 
count. Still others are skeptical that NSF review panels will view improving 
how they provide in-classroom instruction in the discipline or field of their 
NSF funded research as broader impacts. 


This concern has some legitimacy because the NSF itself is not very 
prescriptive in telling review panels how to interpret and assess broader 
impacts. Additionally, NSF review panels are not always well instructed by 
their program officers on what counts and how to effectively evaluate broader 
impacts. Indeed, some faculty members who have included improving in- 
class teaching of undergraduate students as a broader impact have received 
negative feedback from NSF reviewers who have specifically suggested in 
their comments that since ‘teaching’ is a required part of their faculty work, 
activities that help to improve instruction should not be included or counted 
as broader impacts. 


To help to address this concern, AAU successfully added clarifying language 
to the American Innovation and Competitiveness Act (AICA). This legislation, 
approved by the Congress in late December 2016 and signed into law by 
the President January 6, 2017, updated the goals for NSF's broader impacts 
criteria. The new language clearly states that ‘improved undergraduate 
education and instruction’ are means by which NSF researchers can achieve 
a broader impact. 


Building on this change in the legislation, AAU is working though the National 
Alliance for Broader Impacts (NABI), the NSF, other scientific organizations and 
its member institutions to deliver a clear message that the adoption and usage 
by an NSF awardee of evidence-based and/or active and engaged teaching 
practices proven to enhance undergraduate learning and understanding of 
core STEM concepts in disciplines relating on the principal investigator’s NSF 
funded research award should be recognized by NSF review panels as an 
acceptable form of meeting NSF broader impacts criteria. 


Evaluation 


Evaluation is a key component of the Initiative. AAU is assisting member uni- 
versities in tracking the progress of their reform efforts in addition to evaluat- 
ing the overall impact of the Initiative. In this process, AAU has distinguished 
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between measures that are most meaningful at the department level and 
those most useful in documenting cross-institutional effects. 


To support local assessment, AAU has developed a resource guide, Essential 
Questions and Data Sources for Continuous Improvement of Undergraduate 
STEM Teaching and Learning.© The guide provides a set of key questions 
designed to engage institutional leaders and faculty members in discussions 
about teaching and learning. The guide also profiles data sources and ana- 
lytical tools available to answer these questions and inform decision-making. 
It further provides guidance to address shared challenges in evaluating the 
quality and effectiveness of undergraduate education. The Essentia/ Questions 
and Data Sources guide is a complementary resource to the Framework for 
Systemic Change in Undergraduate STEM Teaching and Learning. 


To document cross-institutional effects, AAU collected data from all project 
sites over a three-and-a-half-year period, beginning in the Fall 2013. Common 
data collection included a survey of instructors in participating departments; 
department chair narratives on policy and practice to assess teaching in the 
promotion and tenure process; and campus and department level assess- 
ment of learning spaces [see Online Appendix for project site common data 
collection materials]. 


To begin developing these cross-institutional quantitative measures, AAU 
convened a working group of experts on metrics and evaluation in July 2013. 
Following this meeting, AAU project staff developed a set of research ques- 
tions mapped to the AAU Framework. AAU decided to collect information on 
physical infrastructure (using a portion of the PULSE Vision & Change Rubric) 
and to ask for written descriptions of the role of teaching in promotion and 
tenure by project leads and department chairs. To assess instructor attitudes 
and practices, AAU project staff assembled an instrument from existing tools 
that would be used to survey instructors. Through an iterative conversation 
with individuals at project sites, AAU arrived at a final instrument, as well 
as a collective understanding of how AAU would use the data. Results were 
obtained at two points in time — early 2014 (for the 2013-14 year) and in fall 
of 2016 (for the 2015-16 year). 


Beyond this baseline data request, AAU asked project sites to provide ad- 
ditional information in annual reports. Site-specific data included data on 
student learning outcomes. AAU asked each project site to provide evidence 
for enhanced student learning, but did not require the same indicators or 
metrics to be collected in the same way across the sites [see Online Appendix 
for project site annual and final report requests]. °4 
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Integrated with the collection of baseline measures and annual reports, AAU 
conducted two site visits at each of the eight project sites to allow for a more 
qualitative evaluation of project implementation and progress [see Online Ap- 

pendix for project site interview protocols]. In total, 

AAU met and talked with 325 individuals across the AAU conducted 325 


eight project sites. Teams met with campus project — - . 
leaders, department chairs and deans, and Pro- Interviews across the 


vosts, using these visits to identify challenges and eight project sites. 
possible solutions to implementing project activi- 

ties, and to look at subsequent changes. Site visits also built trust between AAU 

and project sites. While designed for project site campuses, the components of 

this evaluation are useful for any institution interested in assessing its progress. 


In September 2014, AAU announced it had received a second grant from 
NSF, this one to examine the role a national association can play in expand- 
ing reform efforts aimed at improving the quality of undergraduate teaching 
and learning in STEM fields at its member institutions.© This project is being 
conducted in partnership with Adrianna Kezar, Professor, Rossier School of 
Education and Co-Director, Pullias Center for Higher Education at the Uni- 
versity of Southern California. 


Sustaining the Momentum of the Initiative 


AAU found that certain steps taken prior to supporting specific plans for 
implementation were essential to the Initiative’s success. These included 
developing a shared understanding of the Initiative’s goals; collectively 
developing the Framework; agreeing that multiple-strategies are possible to 
achieve the Initiative’s goals; determining how to assess the Initiative’s overall 
impact; and aligning this effort within the national landscape. 


Ultimately the foundational work required to generate buy-in and engagement 
from member campuses in an initiative of this magnitude is critical. Member 
universities need the opportunity for teams to participate in the development 
of a shared vision and framework to ensure that each campus can see their 
local context within the conceptual change model. Time is also critical to build 
the trust necessary for campuses to participate in a collective effort, develop 
a functioning network, and agree to a common data collection effort aimed 
at measuring aggregate impact. 


AAU has successfully established an infrastructure to help align the AAU 
membership toward a common goal. Each AAU member campus has 
designated at least one individual as a campus point of contact for the 
Initiative who acts as a liaison between their campus and AAU on the project. 
To support the Initiative’s activities and goals, the dedicated project director 
situated at AAU engages in continuous dialogue with AAU member campuses 


65 AAU Receives NSF Grant to Study How AAU STEM Teaching Initiative is Producing Educational Change 
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Andrea Follmer Greenhoot, Professor of Psychology, Director of the Center for Teaching Excellence and Gautt 
Teaching Scholar at the University of Kansas, facilitating a knowledge exchange among faculty members around 
evidence-based educational improvement at a TRESTLE meeting. 


along with senior university administrators and designated campus liaisons 
to coordinate and manage the project. The project director also plays an 
essential role in writing grants to secure funding for the various activities in 
support of the Initiative’s goals. 


AAU also continuously brings to the attention of university leadership (President 
or Chancellor, and Provost) results from Initiative activities and successful 
strategies to improve undergraduate STEM education, drawing attention to 
their own campus-based success stories supporting the AAU Initiative (of which 
they are often unaware). AAU recognizes that it is critical to have continuous 
communication and create spaces for relationships and trust among peers 
to develop. At the same time, AAU leadership and staff have also made it 
known to campus leaders when they have observed relative inactivity from their 
particular campus in engaging with the Initiative and/or in focusing attention 
on improving the quality of undergraduate STEM education. 


AAU also collaborates with other national associations, organizations, funders, 
and industry partners to coordinate activities relating to undergraduate STEM 
reform and to develop effective means to disseminate promising and effective 
programs, approaches, methods, and strategies. The Initiative engages 
multiple stakeholders to promote long-lasting reform to undergraduate STEM 
education, and it works to address the cultural and policy barriers within 
research universities that hamper educational improvement and innovation. 
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In 2016 AAU received two major awards. A grant in the amount of approximately 
$1 million over four years from the Northrop Grumman Foundation will 
support institutional mini-grants to further advance and coordinate existing 
efforts aimed at improving undergraduate STEM teaching and learning. AAU 
will award two rounds of twelve mini-grants designed to further existing 
efforts to improve undergraduate education. The first cohort was announced 
in January 2017 and will fund specific improvements in individual departments 
or across colleges at the selected universities. Efforts include creating learning 
communities for STEM faculty members involved in reform efforts, establishing 
programs to train graduate students and undergraduate teaching assistants 
or peer advisors in active learning practices, developing college wide teaching 
evaluation programs, implementing an educational analytics program for the 
university, and supporting STEM course redesigns. 


A grant in the amount of approximately $700K over three years from the NSF 
will allow AAU to examine the institutional landscape in which STEM innovations 
take place to better understand how universities align their various projects to 
promote long-lasting reform to undergraduate STEM education. This project 
led by Emily Miller, James Fairweather and Mary Deane Sorcinelli is designed 
in recognition of the reality that many AAU universities are advancing multiple 
department-level as well as institution-wide efforts to improve undergraduate 
STEM teaching and learning. @ 
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Introduction 


This section presents summaries and analyses of the strategies and approaches 
that the AAU STEM project sites are implementing to improve undergraduate 
STEM education at their institutions. 


This section describes the discipline and curriculum reform focus as well as 
the organizational change models for each of the eight project sites. Looking 
across the various approaches enacted at the eight project sites and their cor- 
responding outcomes, AAU has identified several cross-cutting strategies that 
can be implemented to achieve systemic improvements in undergraduate STEM 
teaching and learning. These cross-cutting strategies are presented in a way 
that aligns them to the Framework along with evidence to support their effec 
tiveness. Throughout the section also are examples and vignettes highlighting 
specific ways in which units are successfully implementing long-lasting changes 
in STEM teaching and learning within their institutional contexts. 


Information from site visits, common data collection, and project site reports 
are among the sources used to provide much of the detailed analysis and 
identification of cross-cutting strategies. 
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AAU STEM Project Sites: 
Change Models and Strategies 


AAU STEM project sites focused from the beginning on the roles of the aca- 
demic department in reforming undergraduate STEM education in large part 
because the AAU Framework and the award process on which it was based 
made clear that the departments are key to effective and sustainable reform in 
introductory STEM courses. This approach is contrasted with the all-too-famil- 
iar approach of working with a self-selected group of reformers which often 
had little effect on colleagues and rarely led to institutionalized reform. The 
central tenet of the AAU Framework is that the key to long-lasting improve- 
ment in undergraduate education lies in the acceptance of and support for 
evidenced-based teaching by the individuals who teach these courses and by 
the senior tenure-track faculty members and department chairs who have 
substantial say in faculty work allocation, in how courses are taught, and 
in faculty rewards. The AAU Framework also required senior administrators 
to pledge additional support (e.g., resources, public expressions of support) 
beyond the project work plan and budget to provide political support for the 
departments attempting to challenge the way that research universities—and 
AAU institutions in particula-—conduct and value undergraduate teaching. 


Before examining the project course and curricular change strategies and the 
roles of the departments, it is worth noting the importance of project leader- 
ship in understanding the nature of the various departmental cultures. These 
range from a senior-faculty member power structure, to power being vested 
in department chairs and deans, to more top-down models with the Provost 
as a driving force. Leadership of project sites designed strategies to fit with 
local cultures when such a fit was key to successful implementation or to chal- 
lenge the local culture when it stood in the way of reforming undergradu- 
ate STEM education. 
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Also of note is the importance over time of promoting and developing a larger 
network of AAU institutions sharing information about undergraduate STEM 
educational reforms on their campuses. It is in part because of this new norm 
of sharing information across peer AAU institutions that some project sites 
were able to successfully reinforce calls for reform by pointing to peer depart- 
ments at other institutions. 


The change models can be categorized into various implicit or explicit mod- 
els of change based on their emphasis on top-down (central administration), 
bottom-up (faculty), or a hybrid (e.g., departmental leadership, faculty learn- 
ing communities) approaches. 


Brown University (Brown) 

The Brown University AAU Undergraduate STEM Education Initiative project 
supports the introduction of evidence-based, high impact practices in key 
introductory courses in physics, chemistry, applied mathematics, and engi- 
neering through the implementation of intensive, small group collaborative 
problem-solving sessions. In these sessions, students work with peers to solve 
conceptually relevant, context rich problems that expose them to interdis- 
ciplinary thinking and learning. The sessions are facilitated by one to three 
individuals involved in the course, such as primary course instructors, depart- 
ment faculty, postdoctoral researchers, graduate students, and undergrad- 
uate students who have previously taken the course. All facilitators receive 
training prior to the first problem-solving session to learn effective ways to 
support student teams. Furthermore, regular meetings of course instructors, 
their department chairs, and Brown AAU Project Team members to discuss 
successes and challenges of the ongoing course implementations have cre- 
ated a best-practices pipeline to quickly disseminate successes and solutions 
between project courses. These meetings also enable cross-disciplinary dis- 
course, nurture departmental communities, and engage faculty and gradu- 
ate students in campus-wide implementation of evidence-based practices. 


Michigan State University (MSU) 

The main goal of the Michigan State University AAU STEM Education Initiative 
project is to bring about change in instructional practices and assessments 
in large gateway courses in order to positively affect student learning. At the 
onset of the project, faculty from biology, chemistry and physics were brought 
together to think about three-dimensional learning, which focuses on the 
core ideas in each discipline, the crosscutting concepts that span science disci- 
plines, and the scientific practices that facilitate students’ use of their knowl- 
edge. These departmental and interdisciplinary conversations engaged faculty 
in developing new assessments, instructional activities, and tools to examine 
the extent to which classroom instruction and course assessments align with 
three-dimensional learning. A new research instrument was created by the 
MSU team to characterize assessments in this manner—the Three-Dimensional 
Learning Assessment Protocol (3D-LAP), and a second protocol to characterize 
instruction — the Three-Dimensional Learning Observation Protocol (3D-LOP) 
— is in progress. Furthermore, a competitive two-year fellowship program has 
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Unlike most of its AAU peers, Brown is primarily a university-college insti- 
tution, with a college existing within the context of a research university. 
The AAU project has stimulated a more critical view of undergraduate 
STEM courses and the role of the faculty in student success. As a result 
of the Initiative, many faculty now view themselves as having a key role in 
whether or not students decide to enroll and/or remain in STEM majors. 


The initial change model at Brown relied on the Sheridan Center for 
Teaching and Learning, which has a distinguished record in promoting 
the use of evidence-based teaching, to lead the project with depart- 
mental involvement limited to some key individuals in those depart- 
ments. The unexpected departure of the Director of the Sheridan Center 
required Brown to change both the structure of the AAU project and 
in its implicit change model. The department chairs and key faculty 
members filled the vacuum left by the Director's departure and assumed 
direct responsibility for the AAU project. Brown has created a permanent 
Faculty Learning Community (FLC) based in departments and colleges 
to promote best instructional practices in STEM across the campus. 
The evidence of increased departmental engagement includes several 
departments (Applied Mathematics, Chemistry, Physics, and the School 
of Engineering) now leading collaboration with the new Brown Science 
Center (rather than relying on the Sheridan Center to make the connec 
tion). Other evidence of departmental increased engagement includes 
the chairs (and the Dean of Engineering)—not individual faculty mem- 
bers—submitting proposals to the AAU project leadership (and to the 
Provost) for AAU-related work. 


been established that supports an interdisciplinary community of faculty who 
are committed to continually improving educational practices in STEM gate- 
way courses by developing new and refined activities, materials, and assess- 
ments that align with three-dimensional learning. 


The University of Arizona (UA) 

The objective of the AAU Undergraduate STEM Education Initiative project 
at the University of Arizona is to change the culture of faculty instructional 
approaches to be more student-centered and use active learning pedago- 
gies. The UA project promotes and supports the redesign of introductory 
STEM courses in three ways: (1) by providing intensive professional devel- 
opment support for faculty to use evidence-based methods; (2) by facilitat- 
ing faculty learning communities (FLCs) among interested instructors; and (3) 
by being a driver for change in making more classroom spaces across cam- 
pus be conducive to active learning techniques. The project team started by 
developing and institutionalizing FLCs to create a core group of faculty sup- 
port for reforming teaching in STEM courses. Along with FLCs, the project 
team created Collaborative Learning Spaces (CLSs) by redesigning existing 
spaces and teaching student-centered, active learning-based introductory 
STEM courses in them. The success of the initial results in introductory STEM 
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CHANGE MODEL 


MSU has embarked on an extensive institution-wide reform effort in 
STEM education. Some of these reforms were put in place prior to or co- 
incident with the AAU project. These actions include hiring highly visible 
tenure-track professors in DBER and science education (including some 
endowed chairs), supporting with university funds the CREATE for STEM 
Institute (CREATE) to coordinate some STEM education activities across 
campus, and using funds from the Provost to implement a program called 
the Biology Initiative which focused on improving undergraduate biology 
education through the investment of new resources in the biology depart- 
ments and Biological Sciences program. The AAU project at MSU began at 
the nexus of these activities, which influenced both its model for change, 
its status at the university, and its strategies. The change model focused 
first on faculty discussions of core ideas of each discipline, and the ways in 
which that knowledge is used, rather than on simply changing pedagogical 
approaches. In this way the intrinsic motivation of faculty is harnessed to 
bring about change. The hypothesis was that faculty who structure student 
learning around core ideas and scientific practices would also, by neces- 
sity change their instructional practices and assessments. MSU’s strategy 
evolved early on to leverage CREATE in coordinating various STEM reform 
efforts (CREATE has since expanded to formally include the College of En- 
gineering and Lyman Briggs College in addition to the Colleges of Natural 
Science and Education) and in supporting various reform efforts both to 
improve the odds for success in the AAU project and to achieve large-scale 
impact at the university. 


Because the change strategy depends on the restructuring of courses 
around core ideas and science practices the evidence of change should 
also incorporate the ways in which both the instruction and assessment of 
student learning have changed. The AAU project team developed a proto- 
col (the 3-dimensional learning assessment protocol 3D-LAP), that allowed 
them to gather evidence of the extent of change in course assessments. The 
3D-LAP also provides guidance for faculty as they design new assessment 
items for transformed courses. Another protocol (the three dimensional 
learning observation protocol 3D-LOP) is under development and will be 
used to gather evidence for the extent of transformation in the classroom. 


As for specific strategies, the MSU central administration invested funds 
in STEM reform (Biology Initiative) prior to the AAU effort. Because the 
AAU project was seen as a cornerstone of an overall reform strategy at the 
university the central administration added support to the AAU team and 
related work prior to evidence of their effectiveness to assist both the AAU 
project leadership and CREATE in coordinating reforms across projects and 
administrative units. Over time the Dean of the College of Natural Science 
and other relevant campus administrative units supported the coordina- 
tion of various large-scale STEM reforms under the CREATE umbrella. The 
success of the AAU project has led directly to a follow on grant from NSF 
to extend the approach to more courses including upper level courses and 
to other institutions. 
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courses resulted in the creation of more CLSs of varying sizes across campus, 
the development of workshops and instructional guides for faculty on how 
to best utilize CLSs to facilitate active learning classrooms, and the targeting 
of FLCs to assist instructors who teach in these spaces. The continual growth 
of CLSs and faculty members using and interested in using active learning 
pedagogies in these spaces indicates a cultural shift in teaching is occurring 
in the desired direction. 


University of California, Davis (UC Davis) 

The AAU Undergraduate STEM Education Initiative project at the University 
of California, Davis works to promote an institutional culture that utilizes data 
and evidence-based pedagogical innovations to make informed educational 
decisions and continually improve classroom instruction and student out- 
comes. Through the creation and implementation of a data analytics infra- 
structure that consists of newly developed tools and visualizations, student 
data and classroom instructional data are being used to inform and improve 
instruction, assessments, curricula, student advisement, and student retention. 
Ongoing course reforms and educational experiments utilizing analytical tools 
are occurring in introductory biology, chemistry, and mathematics courses, 
and the development of new data-based educational experiments has spread 
to other departments. To sustain momentum for these efforts on campus, an 
annual interdisciplinary Scholarship of Teaching and Learning Conference was 
founded for instructors engaged in conducting evidence-based educational 
experiments to share their work. Moreover, the campus project team shares 


The UA change model focused initially on implementing a network of 
faculty members committed to improving STEM undergraduate education. 
Labeled the “coalition of the willing,” UA used a faculty learning commu- 
nity (FLC) model to enable individuals who were often isolated in their 
own programs and departments to work collaboratively with like-minded 
individuals. The FLC provided a mechanism for sharing information about 
and experiences with evidence-based teaching. The FLCbased model also 
was able to add members over time thereby increasing the number of 
faculty members engaged in STEM reform at UA. Since the network was 
supported by and run out of the Associate Provost's office, this approach 
combined a type of bottom-up and top-down model for reform. This ap- 
proach permitted UA to ramp up quickly, using courses taught by members 
of the FLCs as the initial target. The exception was the existing Chemical 
Thinking sequence which had been developed prior to the AAU project; it 
was incorporated into the AAU project in which it has enjoyed a prominent 
place in STEM campus reforms. The combined bottom-up plus top-down 
approach shares some characteristics with the change models at CU Boul- 
der (using Departmental Action Teams as an approximate equivalent to 
FLCs) as well as UNC-Chapel Hill (with a mentor-apprentice model as the 
bottom-up component of the model). 

CONTINUED ON PAGE 54 
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The limitation of the “coalition of the willing” model is that courses tar- 
geted for reform were not necessarily selected because they were linch- 
pin courses or because they taught the most students. They were se- 
lected because members of the FLCs taught them. Again, this approach 
was consistent with rapid implementation and high commitment by 
participating faculty to evidence-based teaching. The challenge going 
forward is that some of the crucial linchpin courses are not taught by 
members of the FLCs nor are they all controlled by departments with 
a substantial commitment to reforms. 


Despite this limitation, the evidence of successful implementation of 
course reforms is substantial. Chemical Thinking has been fully im- 
plemented. Chemical & Environmental Engineering has redesigned 
introductory courses. Molecular and Cellular Biology has agreed on a 
common text and examination. It has fully implemented a reformed 
introductory course. The Physics department agreed on a set of learn- 
ing tasks for revising its introductory course. The current number of 
reformed courses exceeded that which was initially proposed. 


The top-down role of the Associate Provost, who is the Principal Inves- 
tigator for this AAU project site, links the AAU project with institutional 
resources and political support. Importantly, this office also controls 
the use of recently redesigned active-learning oriented classrooms. This 
control ensures that the reformatted classrooms are only used by facul- 
ty members incorporating evidence-based instruction in their courses. 


The FLC model has been effective in providing a formal mechanism 
around which reform-oriented faculty members can coalesce. FLCs 
have gained stature in part because of a deliberate strategy to include 
a substantial number of tenure-track faculty as participants. In addi- 
tion, evidence of effectiveness in student learning outcomes, especially 
course performance in classes taken after the introductory course, was 
leveraged by the campus project team to gain support by departments 
for the reforms. 
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the analytical tools and visualizations they have developed with other inter- 
ested universities. This collaboration helps to refine existing tools and contrib- 
utes to the creation of new tools and evidence-based approaches for improv- 
ing instruction and student outcomes. 


University of Colorado Boulder (CU Boulder) 

The AAU Undergraduate STEM Education Initiative project at University of 
Colorado Boulder targets changes in both culture and structures to foster 
coherent, long-lasting reforms. The project uses a three-layer approach that 
focuses on faculty practices, departmental culture, and administrative sup- 
port/policies. Activities in the multilayers include: (1) working with groups of 
faculty (and often students and staff) in Departmental Action Teams (DATs) to 
create sustainable mechanisms to address educational issues within depart- 
ments; (2) applying targeted approaches to individual departments to stimu- 
late cultural change; and (3) working with senior administration and the fac 
ulty senate to promote the use of evidence-based teaching practices through 
incentives and the adoption of a framework for teaching excellence. The 
three layers are supported by the Office of Institutional Technology and the 
Office of Institutional Research, which provide a common infrastructure across 


Over time, UC Davis shifted from a top-down change model with student 
outcomes data seen as the impetus for reform to a more active strategy 
with project leaders actively participating in the change process. One 
part of this new strategy was the consolidation of several activities into 
a new Center for Educational Effectiveness (CEE), which now combines 
the collection of data about teaching reforms with an administrative and 
policy effort to encourage adoption of new teaching approaches. The 
data collection supported by the CEE continues to focus on institution 
wide and widely disseminated instruments—e.g., ribbon charts on stu- 
dent progress—which enables leaders at the departmental, college, and 
university levels to follow changes in student progress and learning. In 
addition, the CEE increasingly is sponsoring the collection of more com- 
plex data about student learning outcomes (e.g., higher order thinking 
Skills) whose focus is more on instructors and their academic programs. 


These additional data are in part an effort to merge bottom-up, facul- 
ty-led reforms with university-level information about student trends. 


UC Davis does an excellent job of tracking the resources that support 
each of its key activities. The campus project team pools resources to 
meet project (and CEE) goals rather than assigning distinct funding sourc 
es to unique project activities. The AAU project from the beginning has 
targeted linchpin courses for reform—those with the largest enrollments 
and greatest number of majors dependent on them (e.g., General Chem- 
istry). UC Davis has spent considerable time in exporting its data tools to 
other universities—more than 100 other institutions current use either 
the ribbon chart or Generalized Observation and Reflection Protocol 
(GORP) tool. 
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departments for the use of student data and for the collection and use of 
observational data on classroom practices. The project, now branded as the 
STEM Institutional Transformation Action Research (SITAR) project, is housed 
in the Center for STEM Learning and will continue to support educational 
improvements and cultural shifts on campus. 


University of North Carolina at Chapel Hill (UNC-Chapel Hill) 

The objective of the AAU Undergraduate STEM Education Initiative project 
on the campus of the University of North Carolina at Chapel Hill is to support 
the widespread adoption of high-structure, active-learning (HSAL) practices in 
large introductory-level lecture-based STEM courses. The project focuses on 
increasing the number of redesigned, HSAL-based gateway courses in biology, 
chemistry, physics-astronomy and mathematics through the implementation 


CU Boulder’s change model is unique among the AAU project sites. From 
the beginning, CU Boulder focused on institutional and cultural change 
as a method for improving undergraduate STEM education. Change was 
driven by aligning appeals for action from the administration (top-down) 
with on-the-ground faculty work via Department Action Teams (DATs; 
bottom-up) and was supported by infrastructural improvements in class- 
room data collection and student data visualization. This approach relied 
on extensive previous classroom and curricular reforms in STEM at CU 
Boulder through the Science Education Initiative, so it did not include a 
component that engaged faculty members in reformed teaching. This 
approach appeared to lag in its effectiveness through the second year of 
implementation. By the third year, however, evidence suggests that the 
DATs are taking hold and beginning to provide a bottom-up faculty-led 
effort to contribute to the push for reform from the central administration. 


A DAT is a departmental-based working group of about 4 to 8 faculty, 
staff, and/or students that works on a cross-cutting educational issue over 
one or two years, with the support of outside facilitators and the sanction 
of the department chair. DAT participants choose their focus by developing 
a vision for undergraduate education in their department; example foci 
from DATs at CU Boulder include curricular/instructional revision and align- 
ment, improving equity and diversity, and enhancing community among 
faculty, students, and staff. DATs both implement change and focus on 
creating lasting structures (¢.g., committees, positions, policies) that can 
continue their work over time (rather than viewing change as a one-time 
“fix"). DATs maintain transparency by sharing information with and mak- 
ing recommendations for change to the chair, appropriate departmental 
committees, and the department as a whole. Initial DATs at CU Boulder 
initiated a variety of structural changes within their departments, including 
the allocation of several instructor course equivalents to serve as curric- 
ulum coordinators; the formation of a standing committee focused on 
student diversity, retention, and recruitment; and the restructuring of a 
course sequence to better support majors transitioning to upper division. 
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The mentor-apprentice model, the core of UNC-Chapel Hill’s STEM 
reform efforts, has been widely accepted at UNC-Chapel Hill since 
the start of the AAU project because of its demonstrated success in 
improving student learning outcomes, the intentional inclusion by proj- 
ect staff of senior tenure-track faculty members in the apprenticeship 
role (with the mentor being an individual well-schooled in the use of 
evidence based instruction), and the relatively low barrier to implemen- 
tation because of its low cost. Including top researchers in an instruc 
tional reform role has been essential to gaining widespread support for 
reforms by departments and the central administration. Unlike UA, UNG 
Chapel Hill uses FLCs for each participating department which in theory 
is meant to increase the odds of a department adopting STEM reforms. 


Like UA, UNC-Chapel Hill has used data on student success in reformed 
courses to generate political support for institutionalizing the reformed 
courses. This approach has meant using widely accepted metrics when 
available such as concept inventories in Physics. The goal here is to use 
data on student outcomes for both instructors and for the wider depart- 
mental audience. As with UA, collecting data relevant to departmen- 
tal success is crucial. For UNC-Chapel Hill that has meant reducing the 
number of failing grades and withdrawal rates from STEM as much as 
it has meant demonstrating improved student learning outcomes. It has 
also helped the institution demonstrate that reformed courses are sub- 
stantially reducing the classroom performance gap between majority 
and underrepresented-minority students. 


of a mentor/apprentice program that facilitates the transfer of HSAL teach- 
ing techniques across instructors. In the mentor/apprentice program, faculty 
who are experienced in HSAL practices (mentors), and faculty who have less 
experience in HSAL methods (apprentices) work together in pairs to teach 
courses that have been redesigned. After completing the co-teaching expe- 
rience, apprentices go on to teach reformed courses that utilize HSAL prac 
tices on their own. Cohorts of current and past mentors and apprentices also 
meet on a regular basis to participate in department-based faculty learning 
communities. Thus, the project works to create support and incentives, and 
reduce barriers to the adoption of undergraduate STEM teaching methods 
that have been demonstrated to improve student learning gains and close 
achievement gaps. 


University of Pennsylvania (UPenn) 

The project team at the University of Pennsylvania aims to improve introduc 
tory courses in mathematics, chemistry, physics, and bioengineering through 
utilizing teaching practices that foster active learning and student engagement. 
In 2013, the university launched the Structured, Active, In-class, Learning (SAIL) 
program. The program, facilitated by the university's Center for Teaching and 
Learning, assists instructors as they develop, implement and evaluate active 
learning activities to transform their classes into SAIL courses. SAIL courses 
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provide students with the opportunity to struggle through the application of 
course content—often the most difficult part of learning for students—with 
guidance from instructors and help from their peers. Thus, SAIL courses require 
students to do work outside of class time to prepare for in-class activities. The 
Center for Teaching and Learning works to disseminate SAIL practices across 
campus through monthly seminars, provides support for the creation of edu- 
cational videos that can be utilized by students outside of class time, provides 
training for SAIL course Graduate Teaching Assistants, assists in the develop- 
ment of more active learning classroom spaces, and provides financial sup- 
port for faculty reforming their courses through small grants. Assessments and 
the creation of new classrooms have also been an important part of UPenn’s 
change model. The new classrooms create a physical incentive for faculty 
members to assess their own class and rethink teaching. 


Washington University in St. Louis (WashU) 

The project team at Washington University at St. Louis, with support from The 
Teaching Center and the Center for Integrative Research on Cognition, Learning, 
and Education (CIRCLE) promotes the widespread use of multiple active-learning 


UPenn is likely the most faculty-centric institution among the AAU projects. 
The power for change is invested primarily in the collection of individual fac 
ulty members at this type of institution with the role of the central adminis- 
tration seen primarily in terms of resource allocation and political support. 
In this model the unit of change is the individual faculty member in her 
or his classroom. Although many individual faculty members are involved 
in UPenn’s AAU project, representing six departments, project leader- 
ship is primarily invested in the Center for Teaching and Learning (CTL). 
Unlike WashU’s CIRCLE or MSU's CREATE for STEM, the key personnel in 
UPenn’s Center are not tenure-track faculty members. The tradeoff for this 
arrangement is as follows: CTL does not have the same prestige or clout as 
department-led reform initiatives but this arrangement is consistent with 
forming a network of individual faculty reformers when no departmental 
unit has sufficient numbers to form its own collective. To date UPenn has 
instructors participating from Bioengineering, Biology, Chemistry, Earth and 
Environmental Science, Mathematics, Mechanical Engineering, and Physics. 
Overall this approach seems consistent with effective implementation of 
course reforms and dissemination; less clear is the long-term support of 
academic departments and central administration, although the resources 
pledged by central administration to promote STEM reforms to date is an 
encouraging sign. 


The primary mechanism for reform is the SAIL format used as a rubric for 
course reform across departments. The CTL helped establish and coor- 
dinate a STEM Faculty Advisory Board which serves to coalesce interest 
in these reforms across individual faculty members. CTL also serves to 
foster a network for STEM reform by providing Teacher Assistant train- 
ing across programs and departments. 
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strategies in lower-level STEM courses by scaffolding faculty professional devel- 
opment and by creating an environment in which teaching is a community 
effort. The project builds a support structure and collaborative teaching cul- 
ture in which introductory STEM faculty can develop, try out, evaluate, reflect 
on, and refine new active-learning-based teaching methods. Faculty members 
are supported through many opportunities including summer institutes, learn- 
ing communities, reading and discussion groups, a faculty fellowship program, 
a junior faculty mentoring program, and individual instructional consultations. 
Additionally, evaluation studies of pedagogical practices and professional devel- 
opment programs are conducted that create feedback loops to faculty mem- 
bers about their active-learning implementations, and to CIRCLE about their pro- 
grams. These evaluation studies also enable the project team to improve inno- 
vations, foster a wider adoption of active-learning strategies, and contribute to 
current knowledge about the impact of evidence-based methods on student 
learning and effective teaching. & 


Although on paper the WashU AAU project looks similar to that for 
UPenn, the structures and strategies for reform are quite distinct. The 
Center for Integrative Research on Cognition, Learning, and Education 
(CIRCLE) includes tenure-track faculty (like CREATE for STEM at MSU), in 
addition to permanent research scientists, and has more official presence 
with departmental faculty and chairs than CTL at UPenn. As one conse- 
quence, WashU from the beginning has been able to focus on curriculum 
and scaffolding rather than individual course reforms as well as target sus- 
tainability and cultural form. CIRCLE reports to the Provost and is funded 
by hard money from the Provost and Chancellor. 


CIRCLE and the Teaching Center coordinate the Faculty Fellows pro- 
gram and FLCs. It monitors STEM teaching across campus (not just AAU 
reforms). As such it is similar in function to CREATE for STEM at MSU 
(though with a different reporting structure) and the CEE at UC Davis. 
CIRCLE enjoys the support of key departmental leaders which allows 
CIRCLE to operate in more of a partner role with academic units than 
as a support structure (more typical of traditional professional develop- 
ment centers). Like most AAU project sites, WashU has targeted linchpin 
introductory STEM courses for its reform efforts (i.e., Chemistry, Biology, 
Physics). As a result of the AAU project at WashU, institutional funding 
has been secured to target two science departments for reform efforts 
going forward (Biology and Psychology/Brain Science). 
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To document cross-institutional effects, 

AAU collected data from all project sites over 

a three-and-a-half-year period. Common data 
collection included a survey of instructors in 
participating departments, department chair 
narratives on policy and practice to assess 
teaching in the promotion and tenure process; 
and campus and department level assessments 
of learning spaces. 


The instructor survey [see Online Appendix for project site common data collec 
tion materials] focused on: 

= Classroom practices: Instructors were asked to rate how descriptive vari- 
ous statements were of their own teaching practices. 

Attitudes towards teaching: Instructors were asked to indicate their level 
of agreement with statements about teaching practices and techniques. 
Professional development related to teaching: Instructors were asked to 
rate the availability of, and their participation in, various types of on- and 
off-campus professional development activities. 

Institutional environment for teaching: Instructors were asked to indicate 
their level of agreement with statements about the attitudes of other 
instructors, department chairs, and campus administrators toward teach- 
ing, as well as their perception of how important a role teaching played 
in annual and salary reviews and promotion and tenure. 


The survey was administered twice, in 2014 and 2016, to all instructors in 
all participating departments at the eight project sites. Respondent numbers 
and populations varied between the two survey administrations, but in both 
cases more than 60% of respondents were tenured or tenure-track faculty, 
and more than 50% of respondents taught lower-division courses. 


The figure below shows the average response to each survey question about 
practices and attitudes in each survey administration (on a 1 to 4 scale from 
“strongly disagree” to “strongly agree”). Focusing on practices, the range of 
average responses moved from 2.20-3.40 to 2.34—3.51, and overall mean 
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response moved from 2.76 to 2.89. While the differences are not statistically 
significant, none of the items received a lower score in 2016 than in 2014; one 
stayed the same, and 11 of the 12 moved higher. 


Focusing on attitudes, the range of responses moved from 2.83-3.76 to 
2.91-3.75, and overall mean response moved from 3.37 to 3.42. Again, the 
differences are not statistically significant, and most of the shifts are very 
small, but 17 of the 19 items moved upward (one remained the same and one 
shifted downward very slightly). Interestingly, the biggest change in attitudes 
was manifest in the item about inclusivity, a sign that instructors have come 
more conscious of such issues. 


After the first administration of the survey, we identified a gap between 
attitudes and practices: that is, responses to attitudes questions were more 
positive than those about practices that would support such attitudes. The gap 
narrowed slightly in the second survey administration, mostly due to changes 
in practice, but it still exists, suggesting there is more work to be done. Also, 
within the overall sample, there may be important differences between insti- 
tutions, disciplines, and instructor roles on campus (e.g., tenured faculty versus 
lecturers or graduate student instructors) worth additional investigation. 


MeN i=l me) mele]k=xolaal=laien (nated.<lanle leave anlialianlelaanrclalene\icigelel=)) 


with statement about teaching practices and attitudes. 


3.5 


WW 


25 


Practices Attitudes Practices Attitudes 
2014 2014 : 2016 : 2016 


Undergraduate STEM Education Initiative 


61 


Pedagogy 


Extent of Participation 


Direct participation of departments, courses, and students (in terms of stu- 
dent-seats) in the AAU Initiative grew at the eight project sites over the three 
years of the project. At the same time, AAU encouraged other non-AAU proj- 
ect site institutions and departments to undertake reforms. 


At the project sites, the total number of participating departments increased 
from 28 (Year 1) to 37 (Year 2) to 39 (Year 3). The total number of courses 
involved increased from 69 (Year 1) to 143 (Year 2) to 162 (Year 3). The total 
number of student-seats increased from around 38,000 (Year 1) to 45,000 
(Year 2) to nearly 56,000 (Year 3). These numbers are continuing to rise 
because the reforms are expanding to more sections and additional courses. 


The table below depicts the number of faculty members by type of appoint- 
ment engaged in course reforms. 


Faculty Participation 


veal 48 S) 19 5 | 107 
Year 2 89 I) 43 36 190 
Year 3 99 ay, 2) 51 250 


It is important to note that each project site institution had its own plan for 
the project, and those who included more departments and courses are not 
necessarily “better” than those which included less. However, the primary 
goal of the Initiative is to spread evidence-based pedagogical practices as 
widely as possible. 
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Implementation 


All project sites showed evidence of dissemination beyond the initial target 
courses and faculty members. All eight project sites increased the number of 
courses targeted for reform based on evidence-based pedagogy (some expanded 
the number of sections of the same course) and all increased the number of 
faculty members (tenure-track and non-tenure track) participating. One-half of 
project sites expanded their reach to additional departments. One-half of proj- 
ect sites developed and disseminated common tools used to assess teaching 
and instruction, in some cases adopted by the university as a whole. Several 
project sites linked co-curriculuar activities with reformed courses to increase 
retention in STEM majors. 


At Brown, innovative pedagogy formats used during the three years of the proj- 
ect have now became the departmental norm for 13 courses. Instructors of 
some of the introductory physics classes are committed not only to innovative 
pedagogy, but also to assessing students’ learning gains every semester and 
building a longitudinal understanding of how the outcomes of their instruction 
are evolving with time. Also, the number of academic units involved has grown. 
In the Fall 2013 semester, only the departments of Chemistry and Physics were 
involved in the AAU STEM project. Now, the Division of Applied Mathematics, 
the School of Engineering, and the Division of Biology and Medicine and their 
respective staffs are engaged in offering innovative pedagogy to their students 
in introductory and mid-level undergraduate courses. 


One of the key findings from the work undertaken at CU Boulder was the 
need to shift from individual faculty and administrator consultations to depart- 
mental working groups. CU Boulder developed a new model for this work, 
called a Departmental Action Team (DAT).% In the DATs departmental mem- 
bers work collectively, addressing unit-defined issues meant to improve under- 
graduate education in a sustainable manner. The project has facilitated a total 
of six DATs to date, which have often dealt with priority needs specific to each 
department. Examples include creating curriculum coordinator positions to bet- 
ter link sequences of courses, addressing diversity issues, and improving use of 
data in assessments. 


One measure of success of the DAT approach is the expansion from the origi- 
nal departments—Interactive Physiology and Physics—to include Ecological and 
Evolutionary Biology, Mathematics, Electrical and Computer Engineering, and 
Information Science. There is now more interest by departments in forming a 
DAT than there are resources to support that effort. CU Boulder sought and 
was awarded NSF IUSE funds to support the expansion of DATs, both locally and 
at a second partner campus (Colorado State University). DATs will expand into 
Geological Sciences, Atmospheric and Oceanic Sciences, and Computer Science 
at CU Boulder in the fall of 2017. 
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WashU’s efforts have expanded far beyond the original scope of the project, 
including into non-introductory and non-STEM courses. Work has continued into 
a fourth year, with 14 departments/programs, 71 course sections, and 50 faculty 
implementing active learning. All courses that adopted active-learning contin- 
ued to implement it in later semesters, which provides evidence of sustainabil- 
ity. Additionally, one faculty member in the School of Law has expressed inter- 
est in the clicker program, which, if implemented in their course, would expand 
these efforts to a new school and to graduate-level students. 


The UPenn’s SAIL program continues to grow. In departments where individ- 
ual instructors, rather than the department, choose to teach SAIL courses, all 
instructors plan to continue offering their courses in the SAIL format. The Office 
of the Provost, through the Center for Teaching and Learning, has sent out the 
call for more SAIL course development grant proposals, and the number of SAIL 
courses is expected to continue to grow over the next academic year. Some 
departments have expanded their efforts beyond introductory sequences to 
include upper level courses. 


Efforts have also expanded to additional STEM departments and some non- 
STEM departments. For example, the SAIL version of one of the introductory 
economics courses, reaching approximately 600 students annually, is now in its 
fifth semester, and has featured iterative improvements based on student feed- 
back and learning outcomes. This course offers the opportunity to explore ways 
to scale this approach for larger classes, and the number of students allows for 
the evaluation of specific course elements. 


Beyond SAIL, there has been an increase in other active learning efforts. One 
such mechanism is the design of active, student-centered recitations to comple- 
ment larger lecture courses. The Math, Chemistry, and Psychology Departments 
launched versions of their introductory courses with active recitations this year. 
Lessons from the SAIL initiative have been helpful in other endeavors, as instruc 
tors can use previous student feedback on activities and group work to shape 
their approach. 


At UNC-Chapel Hill, a noticeable change has occurred in the culture of the 
departments involved in the AAU project. There is also evidence of spill-over 
effects to other departments. There have been additional course redesign 
efforts in Chemistry and Biology. The Department of Mathematics hired a lec- 
turer knowledgeable in evidence-based methods and has begun to redesign 
its calculus sequence. The Department of Anthropology initiated a depart- 
ment-wide course redesign effort that was influenced by the AAU project, and 
the Departments of Economics and Psychology have continued to expand their 
use of evidence-based teaching methods during the AAU project. Other faculty 
members within the participating units have begun to incorporate many of the 
ideas that have developed from the project. At least one example of a mento- 
ring relationship was established with a faculty member at another university. 
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CASE STUDY 


Structured Active In-class Learning 


Many of the University of Pennsylvania’s activities within the AAU 
Initiative fall under the umbrella of Structured Active In-class Learning 
(SAIL). Broadly speaking, the goal of SAIL classes is actively involving stu- 
dents in doing science and mathematics, rather than watching some- 
one else do it or listening to them talk about it. All SAIL classes include 
three elements: 1) establishing learning goals, 2) in-class active engage- 
ment, and 3) use of out-of-class time (including watching on-line mod- 
ules that contain short videos of lectures and demonstrations and tack- 
ling practice problems before they even arrive in class). 


The university's Center for Teaching and Learning serves as a central 
resource for faculty who teach SAIL classes or are interested in doing 
so. The Center provides both consultations and a formal program of 
support for faculty and TAs interested in this teaching, including a SAIL 
Seminar for faculty and SAIL TA Training. The Center also helps coordi- 


nate instructor access to active learning classrooms on campus. 


In addition, along with the Vice Provost for Education, the Center invites 
faculty to submit proposals for course development grants to support 
the creation of SAIL classes. SAIL grants provide faculty with $5,000 for 
their preparation time or for graduate student assistance in the process 
of developing in-class exercises, any out-of-class materials, or assign- 
ments and assessments. As of the 2016-2017 year, there have been 13 
SAIL grants in 11 different fields. 


The number of SAIL courses continues to grow. In 2016-17, the univer- 
sity had 27 SAIL courses, including eight in non-STEM fields. This is an 
increase from nine total courses in 2013-14. Likewise, the number of 
instructors who teach SAIL courses increased from 12 to 33 during the 
same interval. Overall, 51 unique instructors at UPenn taught or will 
teach SAIL courses between 2014 and the summer of 2017, and num- 
bers of both courses and instructors are expected to continue to grow. 


UC Davis has expanded its efforts in several departments. Introductory biol- 
ogy underwent a major and long-lasting change as a result of a combined 
Gates Foundation and AAU effort to improve student outcomes. Teaching 
assistant training resulted in a highly structured discussion environment 
that emphasized group work and high order cognitive skills development. 
Combined with a modeling-focused active lecture, students are now able 
to successfully complete substantially higher order problems, as based on 
Bloom's taxonomy. In addition, the changes in the introductory course are 
having measurable effects on subsequent course performance. 
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In the Chemistry Department the introductory chemistry restructuring project 
expanded substantially beyond the rethinking of the initial course sequence. 
Three major components emerged including: 1) the methodical measurement 
of learning as a function of the instructional approach for the introductory 
courses, 2) a major overhaul of the chemistry preparation pathways, and 3) 
an entirely new three quarter sequence for life science majors. 


For the first component, it was shown that instructional approach could be reli- 
ably measured using the Classroom Observation Protocol for Undergraduate 
STEM (COPUS) instrument via the Generalized Observation and Reflection 
Platform (GORP) tool, which was developed by the CEE team.®’ Additionally, 
it was found that active learning instructional approaches yielded improved 
learning compared to traditional approaches, especially with less prepared 
student populations. 


For the second component, the Assessment and Learning in Knowledge 
Spaces (ALEKS) preparatory chemistry adaptive learning summer self-paced 
course was tested, replicated, and fully incorporated to help prepare students 
for immediate entry into the chemistry sequence. In summer 2016, over 600 
students could enter the chemistry sequence without needing a three-unit, 
non-credit bearing, preparatory course in their first quarter. This approach 
better prepares students, saves them time and money, and improves initial 
chemistry course success. The overall success of this program has inspired the 
Math Department to try a similar approach. 


For the third component, this approach focused on conceptual knowledge 
and life science application with an entirely revised laboratory that empha- 
sizes open-ended problems. The Chemistry Department chose to adapt the 
UA Chemical Thinking curriculum for several of the courses and currently is 
examining the impact. 


MSU's AAU project focused primarily on the main lecture sequences in gen- 
eral chemistry, introductory biology, and introductory physics. However, other 
course committees and instructors have begun to use the three dimensions 
developed by MSU (scientific practices, crosscutting concepts, and core ideas) 
and adaptations of the three dimensions as the framework for transformation 
efforts. These courses include a pre-general chemistry course that focuses on 
connections between chemistry and mathematics, organic chemistry, the gen- 
eral chemistry laboratory sequence, one section of an upper-division physical 
chemistry course, second-tier yet foundational biology courses including ecol- 
ogy and evolution, calculus for life science students, and introductory phys- 
ics for life science students. 


The goals of MSU's Biology Initiative and its AAU project overlapped sub- 
stantially, including an overall focus on scientific practices and core ideas in 
courses and degree programs. Additional TAs and LAs provide the resources 
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necessary to facilitate more student-centered and active approaches in class 
meetings as well as implement more frequent assessments and feedback. 
Curriculum coordinator positions have also been implemented for several 
courses, to develop and maintain a shared vision for the courses based on a 
core set of learning goals and to develop and evaluate new course materials 
and to coordinate assessment of student learning across sections. 


In the Chemistry Department, implementation of transformed courses led to a 
permanent increase of resources including the creation of two new permanent 
instructional faculty positions: a lecturer for general chemistry and a Director 
of General Chemistry Laboratories. These positions allowed the Director of 
General Chemistry to focus on the transformation effort, coordinate the new 
materials, and provide support for faculty who rotate into the new course. 


The success of the transformed course in physics has led to faculty support 
for expanding the course to additional sections, piloting a second-semester 
course, developing an integrated lecture-lab model for those students hav- 
ing to take both, and further development of the transformed laboratory. The 
Physics faculty voted overwhelmingly to embrace these changes to the extent 
that the physics budget can support them. The department has committed to 
supporting a single instructor line that is devoted the expansion and continued 
development of the transformed course, and additional resources for devel- 
opment of both transformed lecture and laboratories have been endorsed by 
the faculty and requested from the college and Provost. 


All of the UA’‘s originally proposed course reforms were expanded within tar- 
get departments. Additionally, in chemistry, the work on Chemical Thinking 
has expanded to: 


= The development of a version of the curriculum for Honors students. 


= The creation of a pilot preparatory course to better support students with 
weak academic backgrounds before they enroll in Chemical Thinking. 


= One faculty member in Organic Chemistry using active learning instruc 
tional approaches and teaching in one of the Collaborative Learning Spaces. 


= The use of the Chemical Thinking Curriculum at other three other 
U.S. universities. 


In physics, seven different instructors have used active learning to teach one intro- 
ductory class. The active learning, student-centered teaching approach has been 
taken in five other physics courses, with reform of a sixth beginning this fall. The 
department has also started its own Faculty Learning Community focused on 
improving instruction and student learning in undergraduate physics courses. 


Evidence Related to Student Learning 


Each project site included a plan for assessing the effectiveness of the class- 
room interventions they proposed to use or test. These varied in form from site 
to site. AAU encouraged individual sites to use these data to inform ongoing 
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practice, and to publish results as appropriate. AAU did not ask the sites to 
report learning outcomes data in a form that could be aggregated as with the 
responses to AAU’'s instructional survey. Summarized here are results shared 
in the sites’ annual reports that point to trends across the Initiative toward 
improved learning gains, decreased failure rates, improved persistence from 
introductory to later courses, and shrinking achievement gaps for previously 
underserved students. 


Across the project sites a variety of tools were used to measure content or skill 
mastery. These included validated concept inventories developed by academic 
disciplines, inventories developed locally, new assessments developed following 
departmental adoption of learning goals, common exams administered across 
multiple sections of a course, and course grades. In addition to measures of 
content and skill mastery, most sites tracked additional student outcomes indic- 
ative of improved instruction often in line with institutional priorities. 


Although many of the interventions are still in process with data gathering 

and analysis at an early stage, some initial (and substantial) trends are evi- 

dent. Every site reported some improvement in student learning outcomes. 

The magnitude and significance varied according to the different stages of 

the reform process across the institutions and across departments in some 
institutions. Dramatic reductions in achievement gaps especially for women, 
under-represented minorities, and first-generation 

students were observed in some sites. Reports of Project site outcomes 
decreased DFW (D grades, F grades, and withdraw- . 

als from a course) rates were common, as were documented In 42 
increased persistence to the next or later courses beer reviewed articles. 
and success in later courses as measured by grade 

performance. Improved performance on exams sponsored by disciplinary soci- 

eties was observed, as was stronger performance on disciplinary concept 

inventories. Some sites also have tracked the effects of instructional inter- 

ventions on more general psychological factors, such as self-efficacy, meta- 

cognition, and student attitudes toward science. 


MSU included an explicit strategy for changing the ways that student learning 
is assessed. Use of the 3D-LAP allowed the project team to explicitly identify 
assessment tasks that require students to use their knowledge in the context of 
scientific practices. Early research results from MSU's work are very promising, 
and provide the team with insights about factors that affect transformation of 
large scale courses. Research studies on student learning in chemistry show that 
compared to students in traditional courses, students in the transformed courses 
show significantly increased understanding and use of core ideas in chemistry. 
This, coupled with a decrease in the rate of DFW, on average, around 450 more 
students per year now pass the course with a grade of C or better. 
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John Pollard, Associate Professor of Practice, teaching a Chemical Thinking course at the University of Arizona. 


At UNC Chapel Hill the academic year average D/F rates in redesigned courses 
dropped from 11.5% in 2013 prior to the AAU project to 9.5% in 2016. The 
learning gains in HSAL courses were 13% higher than in traditional courses. 
Teaching Assistants at UC Davis trained to use active learning practices and 
adaptive learning technology from Carnegie Mellon were able to raise stu- 
dent outcomes by half a letter grade (and/or increase the probability of pass- 
ing exams by 66%) in introductory biology. 


As a result of the Initiative at WashU, 57 STEM faculty, from 9 of our 11 STEM 
departments, have taught 49 different courses (76 course sections) using per- 
sonal response systems (i.e., “clickers”). WashU’'s evaluation studies found 
that the clicker-based active learning used in three of the courses, which were 
high-enrollment introductory science courses, was positively associated with 
exam performance, even when students’ cognitive characteristics (ACT, AP test 
scores, scores on course pre-tests) and class attendance were accounted for. 


CU Boulder’s Departmental Action Teams each worked toward department level 
consensus on learning goals, pedagogical approaches, and assessments aligned 
with the learning goals as a way to build widespread and lasting change. In the 
physics department, CU Boulder conducted pre/post conceptual assessments of 
four courses (Calculus-based General Physics | and Il and Algebra-based General 
Physics | and Il) over five semesters of the AAU project. The results averaged 
across semesters indicated that students from all four courses had higher post- 
test scores between 25% and 30% in reformed courses. 
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UA found improved learning gains in sections of physics with calculus taught 
with active learning approaches, compared to the traditional lectures class. 
The reformed course final exam scores outperformed the traditional course 
on every exam item. The redesigned Chemical Thinking has demonstrated 
that students’ performance on the American Chemical Society standardized 
exam were not significantly different from those in the traditional course. 
However, students in Chemical Thinking performed significantly better on a 
conceptual chemistry questionnaire (55.3% vs. 44.3%) and had significantly 
more positive attitudes toward chemistry than the traditional group. 


Not surprisingly, since the Initiative overall intended to catalyze change, much 
of the assessment work was used formatively over the several years of the 
interventions. Reports included multiple accounts of fine tuning of approaches 
based on student outcomes, with significant growth and insights into fac- 
tors that make a difference in impact such as student readiness and inter- 
est, physical settings for instruction, whether students are taking the course 
for the first time or repeating it, and effective practices for managing group 
dynamics for group work. UC Davis tested among other things the use of free 
online textbooks in two disciplines, and found students did as well with them 
as with traditional textbooks (a significant financial benefit for the students). 


The first published reports of specific impacts can be found in the “AAU STEM 
Scholarship” section on page 102. Hf 
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Pedagogical Expertise 


One strategy to successfully institutionalize reform is to embed instructional 
expertise within the department to provide educational leadership and to 
support all faculty members in the adoption and use of evidence-based ped- 
agogy. Although the types of appointments of individuals with this type of 
expertise vary widely (e.g., tenure-track, non-tenure track, junior and senior 
ranks), these individuals all have in common an understanding of the disci- 
pline and how students learn best within the discipline. When used most 
effectively, these individuals are well positioned to provide educational lead- 
ership to the department. 


Individuals in these roles can help redesign courses, co-teach courses with 
other faculty members, take on the responsibility for departmental-wide edu- 
cational improvement, and produce scholarship for the broader higher educa- 
tion community based on the evaluation of reformed courses. These individu- 
als often will also engage in efforts with other STEM departments to achieve 
broader institution-wide systemic STEM teaching reforms. Acceptance and 
support of individuals with instructional expertise—even those in tenure-track 
positions—by departmental leaders and by tenure-track faculty members 
is also an essential element for long-lasting change. In Biological Sciences, 
Physics and Astronomy, and Chemistry, The UNC-Chapel Hill has made several 
teaching-related hires with backgrounds in evidenced-based STEM teaching 
since the start of the AAU project. Over time, these personnel have been seen 
by chairs and their faculties as strategic hires whose help in achieving over- 
all departmental instructional objectives is paramount. UNC-Chapel Hill staff 
(and AAU staff) have found these hires fully entrenched within the respective 
departments and are beginning to have “lasting impact on the teaching cul- 
ture within departments. As another example, MSU hired a curriculum coor- 
dinator for introductory biology courses and added staff to develop inqui- 
ry-based laboratories in Chemistry. As a third example, WashU is embedding 
educational specialists in STEM departments. WashU also hired a coordina- 
tor for General Chemistry and appointed a new Vice Dean for Education in 
the College of Engineering. 
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Mentor-Mentee Approach 


The University of North Carolina at Chapel Hill has embedded pedagogical 
expertise within departments by capitalizing on their faculty track of lectur- 
ers. Departments have used these positions to hire individuals who bring 
educational expertise and leadership into departments. Such lecturers, who 
are eligible for tenure, have experience in evidence-based teaching meth- 
ods in the sciences and are evaluated on their ability to contribute to the 
ongoing evolution of the departmental and institutional teaching culture. 


The presence of teaching-oriented faculty who are treated with status 
equivalent to more research-oriented faculty has lowered the barrier for 
entry for all faculty in departments to think about how to infuse evi- 
dence-based pedagogy into their classes. One way in which this knowl- 
edge has been disseminated is through mentor-apprentice relationships 
between teaching-oriented and other faculty, including senior tenured 
faculty. Departments have incentivized these relationships by giving both 
faculty members credit for teaching the course they are working on 
together that term. So far in the project, 27 such “course releases” have 
been provided. Going forward, the Dean of the College of Arts and 
Sciences has committed to supporting four course releases per year for 
course redesign and apprenticeships for three years. 


Michael Crimmins, a senior professor of chemistry, participated first as 
an apprentice and later as a mentor. Crimmins undertook a complete 
reform of the Introductory Organic Chemistry 1 course into a high struc 
ture, active learning format. 


Inside Higher Ed quoted Crimmins: “We're trying to get students to 
engage with content multiple times, to get some information before 
they come to the classroom. Then I’m a lot less of a lecturer than | used 
to be. ... The huge change is that | don’t walk into class with notes or a 
PowerPoint -- | have some notes -- but it’s not me talking, talking, talking. 
| talk a little bit but I’m posing students questions, and they're talking 
among themselves or in groups.” 


After redesigning the course, Crimmins observed that failure rates had 
been reduced for all students (from 17% to 6% when comparing the 
2002-03 to 2013-14 classes), but notably for African American students 
(from 33% in 2002-03 to 14% in 2013-14) and all underrepresented 
minority students (from 21% in 2002-03 to 11% in 2013-14). 


Other courses at UNC-Chapel Hill, including in the biology department, 
witnessed a similar closing of the achievement gap for both first-genera- 
tion and African American students. Additionally, evidence shows more 
women are now enrolling in gateway science courses at UNC-Chapel Hill. 
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Individuals hired to enhance the instructional expertise of their programs 
and departments—even tenure-track discipline-based educational research- 
ers—can find acceptance by their departmental colleagues difficult. Even 
though UC Davis is part of the University of California System, which has a 
designated employment category for teaching-oriented positions endowed 
with the possibility of tenure (LPSOE - lectures with the potential of secu- 
rity of employment), some of its STEM departments have found it diffi- 
cult to fully utilize and accept LPSOE hires. Cultural differences between 
departments make it quite difficult to see uniform change across UC Davis 
in this respect. The Center for Education Effectiveness (CEE) is working with 
departments and colleges to clarify promotion and tenure criteria and cre- 
ate tools to help all faculty members including LPSOE hires document their 
evidence-based practices. 


In addition to adding positions with specific teaching expertise, enhanc- 
ing the value placed on teaching at research universities starts with 
stated expectations for teaching when new tenure-track faculty members 
are hired. At UNC-Chapel Hill, for example, new STEM hires (including 
those with tenure-track positions) must have experience in the use of evi- 
dence-based teaching and be prepared to participate in the mentor-appren- 
ticeship model. Also, MSU substantially enhanced the visibility of STEM edu- 
cational reforms by hiring several tenure-track endowed chairs focused on 
STEM educational reforms. 


Institutional Data Analytics and Visualizations 


Data analytics and visualizations played a role in most of the project sites. 
Data focused on both faculty practices and student outcomes, the latter 
including academic performance and progress toward the degree. Much data 
collection and visualization about faculty instructional practices focused on 
classroom observation protocols. 


UC Davis developed the Generalized Observation and Reflection Platform 
(GORP), which has been used with the Classroom Observation Protocol for 
Undergraduate STEM (COPUS) protocol to observe interactions between stu- 
dents and faculty in large STEM classrooms.® This tool has been disseminated 
through the Tools for Evidence-Based Action network and is now in use by 
over 100 universities.’”° 


At CU Boulder, classroom observations were conducted with modified ver- 
sions of the Teaching Dimensions Observation Protocol (TDOP) and COPUS.”' 
CU Boulder’s classroom observation protocol has been branded as VIP/OPLE 
(Visualizations in Instructional Practice & Observation Protocol for Learning 
Environments). The university's Office of Information Technology is working 


69 UC Davis GORP Tool and COPUS Protocol 
70 Tools for Evidence-Based Action 
71 Teaching Dimensions Observation Protocol (TDOP) 


Undergraduate STEM Education Initiative 


75 


76 


CASE STUDY 


Departmental Action Teams 


To embed pedagogical expertise within departments, the University of 
Colorado Boulder developed a new type of faculty, staff, and student work- 
ing group called a Departmental Action Team (DAT). DATs support mem- 
bers of a department in identifying an educational issue of broad-scale 
importance that they want to address and in making sustainable changes 
by designing and implementing new structures and by shifting departmen- 
tal culture to address the issue. 


A DAT consists of a self-selected group of about 4-8 participants and an 
external facilitator; the departmental participants are primarily faculty, but 
may also include postdoctoral researchers, undergraduate and graduate stu- 
dents, and staff. DAT members select an educational issue of shared inter- 
est within their department and work collaboratively to address it. A DAT’s 
focus goes beyond individual course redesign to broader issues. Moreover, 
the team works collectively, which distinguishes a DAT from a similar struc 
ture, the Faculty Learning Community. 


DAT facilitators bring expertise in educational research and institutional 

change, help coordinate logistics, and help the group to work together 

in a collaborative fashion. Additional incentives (such as service alloca- 

tion, teaching credit, and other recognition) ensure that participation in 

the DAT is not a wholly volunteer act, but rather is itself built into the 

departmental structure and reward system. Thus far, six DATs have been 

facilitated, pursuing a range of different goals. For example: 

= The Mathematics Department's DAT focused on integrating core courses 
for majors by collecting and analyzing data related to these courses. As 
the result of this work, the department has developed a new course for 
majors that it will begin piloting over the coming year. By revisiting stu- 
dent outcome data, the DAT will assess the impact of the course. 

= In the Physics Department, the DAT successfully created an ad hoc com- 
mittee to address issues of diversity. Among other activities, this com- 
mittee is working to redesign the department website to better serve 
and recruit underrepresented students, taking steps to encourage fac 
ulty mentoring, and creating regular opportunities for members of the 
department to engage in conversations and self-education around issues 
of equity and inclusion. The DAT is now a standing committee in the 
department, using data to guide decision-making, and will continue to 
collect data about student success (broken out by different demographic 
groups) to understand the impact of its work. 
The Information Sciences Department was newly formed in fall of 2015. 
Its DAT worked to develop its undergraduate curriculum and set up basic 
procedures for operation. Backwards design techniques were used to cre- 
ate the curriculum for this new major, starting with big picture goals and 
mapping them backwards into specific sequences and courses. 
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on a scalable version of the TDOP that will provide faculty, on a voluntary 
basis, with observation-based visualizations of what happens across class 
periods.’* This service should benefit faculty members interested in gaining 
new insights into the patterns of their class activities, documenting changes 
as they try out new methods of teaching, and finding new ways to commu- 
nicate about their teaching with colleagues. 


WashuU is studying faculty use of clickers using both a quantitative observa- 
tion tool, the Observation Protocol for Active Learning (OPAL), and qualita- 
tive interviews. Observational data are analyzed to identify patterns and clas- 
sify instructors into four categories based on how they used active learning 
with clickers in their classrooms.’? OPAL, which was developed as part of the 
AAU Initiative, has been institutionalized at WashU by the Teaching Center. In 
addition to the 200+ classroom observations made as a part of AAU research 
studies, the Teaching Center has integrated data from the tool into faculty 
consultations. A visual timeline of the data can be generated, providing fac 
ulty with an overview of the activities that occurred in their classroom. The 
timeline data can be used in addition to a more traditional classroom observa- 
tion by a faculty developer, an approach WashU calls Multimodal Observation 
for Scholarly Teaching (MOST).” 


MSU has developed the Three-Dimensional Learning Assessment Protocol 
(3D-LAP), a set of criteria that can be used to determine if an individual 
assessment item (or cluster of items) aligns with a scientific practice, cross- 
cutting concept, or disciplinary core idea (the “three dimensions”).”> They 
have used the 3D-LAP to characterize almost 5,000 assessment items over 
the course of the AAU project at MSU, fully representing all 200+ course 
sections of the eight relevant gateway lecture courses in biology, chemistry, 
and physics. 


Institutions have looked at a range of student-level information. UC Davis 
developed Ribbon, which allows departments and campus administrators 
to look at migration in and out of STEM departments and between depart- 
ments.’ More than one-third of the STEM departments and several non- 
STEM departments now use the tool with many faculty members and profes- 
sional advisors using it repeatedly. By 2017, more than 100 universities nation- 
wide and internationally were using the tool to visualize student migrations as 
part of the Helmsley-funded Tools for Evidence-based Actions (TEA) project. 


As the work of the Center for Educational Effectiveness (CEE), much of which 
is related to reforms begun under the aegis of the AAU project, has become 
known across the UC Davis campus, additional departments have started 
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Harnessing Institutional Data 


UC Davis has long been a leader in harnessing institutional data to identify, 
measure, and research factors that promote or inhibit the effectiveness of 
undergraduate education. Many of these efforts are currently centralized 
through the university's Center for Educational Effectiveness (CEE). 


Part of this leadership has come through devising novel ways to get 
traditionally separate types of university data to work together to pro- 
vide a fuller picture of the student experience. Another part comes from 
the development of powerful and intuitive data visualizations to display 
this information. Ribbon, for example, allows departments and campus 
administrators to look at migration in and out of STEM departments and 
between departments. More than 100 universities worldwide are currently 
using some form of this tool to visualize student migrations. CEE has also 
developed a Heat Map tool to identify courses with large DFW (drop, fail, 
withdraw) rates or courses that lead to disproportionately poor perfor- 
mance in subsequent courses. 


CEE continues to develop tools for faculty and administrators. Many of 
the visualizations will soon be incorporated into the new campus Tableau 
server to facilitate secure delivery to department users. Know Your 
Students (KYS), for example, is intended to help faculty with their large 
courses by: 1) providing information about students’ backgrounds and 
tips for engaging different types of students prior to the first day of class, 
2) collecting and presenting targeted information on student out- comes 
during the course, and 3) aggregating, analyzing, and cataloging course 
outcomes in a teaching “portfolio” for reflection and use in the merit and 
promotion process. The first stage of pre-course information has been 
tested and iterated while the other two stages (MIDAS - Multidimensional 
Instructional Development for Achievement and Success) have received 
HHMI funding to develop. 


The Diagnostic Department Dashboard (DDD), is meant for department- 
and campus-level administration and displays and helps interpret infor- 
mation about enrollment, diversity, retention, course “hot spots”, student 
performance gaps, instructional staffing and more. This tool has been 
used formatively to assist various academic units in developing plans to 
improve student outcomes. 


Beyond developing visualizations and tools, CEE has taken a strategic 
approach to using this information to bring about action. For example, 
DDD has been used in consultation, rather than being made independently 
available, to avoid misinterpretation and provide an opportunity to suggest 
relevant support services. 

CONTINUED ON PAGE 79 
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courses and programs to inform department decision making. 


The Ribbon Tool developed by CEE at UC Davis visualizes undergraduate students’ educational pathways through 


collaborations with this unit. For example, the College of Agriculture has 
engaged in multiple studies of undergraduate student outcomes in programs 
including animal science, human development, environmental management, 
and agricultural economics. These studies focus on pre-requisites, trouble- 
some large introductory courses, longitudinal course outcomes, retention 
of under-represented groups, development of course and program learn- 
ing objectives, and online/hybrid course development. The departments of 
Communication, Economics, and Psychology have been involved in the devel- 
opment of hybrid and online courses with expanding plans for instructional 
research in the 2017-18 academic year. 


The CU Boulder has undertaken an initiative called Data Analytics for Student 
Success and Educational Effectiveness (DASSEE). The tools developed in this 
project make it possible for departments and administrators to track impacts 
of courses and pathways of students across courses and address campus 
priorities for improving student retention, and diversity. Using Tableau as a 


CONTINUED FROM PAGE 78 


Underlying this approach is the understanding that while data by them- 
selves may not change behavior, they can serve as the basis for conversa- 
tions that will. Visualizations can help focus these conversations. Data must 


be understood in institutional context, and often more local context as 
well. While larger studies can highlight overall problems, issues, or trends, 
localized data (e.g., data about a single department or set of courses) is 
more effective and in influential in terms of bringing about specific action. 
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visualization platform, queries of the student information system can be per- 
formed focusing on a single course over time, a suite of courses in a given 
department or unit for a given semester, or a dashboard of individual stu- 
dent course grades by major over term. Key visualizations in the dashboard 
include representation of enrollment, grade distributions, pathways into a 
given course and following course, retention/persistence of students from 
a given course over time (in the major and institution). Subgroups can be 
selected by term, faculty instructor, demographics (including: an individual 
instructor offering of a course, gender, first generation status, ethnic/racial 
identification), and for subgroups receiving a given grade in a course. 


Efforts to visualize student data have involved partnerships with Institutional 
Research (IR) and the Office of Information Technology to implement visu- 
alizations in Tableau. The goal is to create a suite of easily accessible data 
sets and associated visualizations of educational outcomes for undergrad- 
uates. Although these data already exist, they are not available in an easily 
accessible way. Similarly, at the UPenn, the Office of Institutional Research 
& Analysis (IR&A) is working with the Center for Teaching and Learning 
to establish a protocol for data requests from instructors assessing their 
courses. By developing assessment tools in the Center for Teaching and 
Learning and making them available to course instructors, UPenn has 
reduced the “time barrier” and encouraged more faculty members to use 
evidence-based instruction. 


Keys to Successful Use of Data Analytics 


|. Distinguish between the types of data useful for individual faculty mem- 
bers designing and assessing their courses from the types of data used in 
departmental decision-making. The key data influencing the faculty mem- 
bers (and chair) in the Chemistry and Molecular Biology Department at 
UA to adopt Chemical Thinking as the preferred introductory sequence 
in General Chemistry were the success of these students in subsequent 
courses; detailed within-course data on student learning outcomes 
demonstrated in the course were not important in the departmental 
decision. As another example, UNC-Chapel Hill and UPenn demonstrated 
decreases in Ds and Fs and course withdrawals from reformed courses. 
Departmental and university leadership were very interested in such 
data, particularly when underrepresented minorities showed differential 
improvement on these metrics. 


2. Ease and efficiency of use are keys to broad acceptance of teaching-re- 
lated metrics. CU Boulder DATs have emphasized commonly-used met- 
rics on student outcomes, which makes dissemination and use in deci- 
sion-making more likely. Linking IR into the data sharing process is also 
important. 


3. Data must be seen as part of the policy and decision-making process 
rather than as sufficient in their own right. Among the more important 
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lessons learned on the use of data in educational reform took place at 
UC Davis. Initially the data analytics developed by UC Davis—which have 
broad appeal nationally and are quite widely used—were assumed to be 
sufficient to show faculty and administrators what worked and why. Over 
time it became clear to the CEE that passive presentation of data were 
not effective in promoting the redesign of courses and curricula. Rather, 
the CEE had to change its role to work more actively with academic units 
to devise strategies based on data analytics. 


Collaborative Learning Spaces 


The University of Arizona has focused some of its efforts to redesign 
classrooms into collaborative learning spaces (CLSs). Thanks to these 
recent efforts, the university currently has ten CLSs ranging in size from 
30 to 264 student-seat spaces, and plans to redesign ten additional 
classrooms into CLSs ranging in size from 25 to 135. 


One faculty member, Dr. Zoe Cohen from the Department of 
Immunology, documented her thoughts about participating in the AAU 
STEM Initiative and teaching in CLSs: 


“| had been thinking about trying a “flipped” classroom for a while and 
decided to take the plunge and apply for one of the rooms. | was lucky 
enough to get accepted and spent the summer preparing for a com- 
pletely new form of teaching. | taught Physiology of the Immune System 
(PSIO 431) to 160 students in the Science and Engineering Library in 
Fall 2015, and this experience not only changed me as an educator, it 
has opened opportunities to me to continue along this pathway. Once 
| started teaching in this room, | became involved in a Faculty Learning 
Community regarding the CLS as well as an Educational Faculty Learning 
Community. Through these, | was able to develop the materials to incor- 
porate active learning (through discussions) within the classroom set- 
ting .... Based on teacher course evaluations and other comments, as 
well as looking at the overall grade distribution in the course (compared 
to previous semesters when the course was taught in a traditional lec 
ture style), the changes made resulted in more active and meaningful 
engagement and understanding by the students.” 


Prof. Cohen has seen more As and Bs and fewer Ds and Es on the final 
exam since transforming her Physiology of the Immune System course. 
Her experiences match those of other faculty for different classes and 
departments (including physics, chemistry, molecular and cellular biol- 
ogy, and engineering), who also have seen positive impacts on student 
learning. These impacts include better grades and fewer failures, stron- 
ger performance on pre-post course tests, and better performance in 
follow-on courses. 
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AAU Learning Space Data 


AAU collected data on campus infrastructure, 
including learning spaces, from project sites 
using the relevant sections of the PULSE Vision 
& Change Rubric. Given the expense and time- 
consuming nature of reconfiguring classroom 
spaces, AAU did not expect to see significant 
change in physical space over the three years 
of the project. However, institutions have made 
some progress in redesigning learning spaces. 


The table shows the average aggregate response of institutions to each 
item during the first (2014) and second (2016) data collections, as well as 
the average aggregate response of Provosts during the second data collec 
tion. Interestingly, of the ten items, eight scored /ower in 2016 than 2014. 
There are a few possible explanations. One is that these changes are due 
to differences in the sample. While comparisons between the first and sec 
ond data collections are interesting, it is important to keep in mind that 
different departments are being compared, as well as different numbers 
of departments for each institution. Another possible explanation is that, 
over the course of the project, there were changes in attitudes leading to 
higher expectations. These self-reported rubric scores, especially when com- 
pared across time, may be at least as indicative of attitudes as they are of 
physical change. 


The table also looks at differences between departmental and Provost 
views. The table shows that departments were more critical of learning 
spaces and resources and support; Provosts were slightly more critical with 
respect to physical infrastructure. The table also shows that for all but the 
last item (Institutional support for electronic resources: e.g., journal sub- 
scriptions and databases) there was a great range of differences in views 
across the eight universities. 
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2014 2016 2016 


Fa Ele r Aggregate Aggregate Aggregate 


Response Response Response 
(Depts.) (trey oy 9) 


Physical Infrastructure 


Classrooms and teaching laboratories can accommodate 
special needs and differing abilities. 3.7 2.9 3.3 


Access to flexible, re-configurable teaching spaces to encourage 
student interaction, ability to work in small groups. 2:5 1.8 2.0 


Classroom IT infrastructure encourages active learning practices. 2.8 3.0 2.4 


Access to intelligently designed laboratory space flexible enough 
to allow different uses that blur distinction between lecture and lab. 20 2.5 1.8 


Equipment/supplies in teaching laboratories. 2.7 2.3 2.1 


Learning Spaces 


Informal gathering spaces that encourage collaboration. 2.5 2.2 2.8 


Learning Center for Students — for example, college-wide writing 
centers, learning centers or department level center with staff, tutor 
meeting rooms, TAs, computers and printers, study space for students. 3.2 2:5 3.3 


Resources and Support 


IT support for innovative teaching, responds quickly to IT crisis; 


support includes hands-on technology training for faculty and 
proactive survey of new technology. 3.4 3.0 3.3 


Staff support for teaching: administrative help to support teaching, 
lab managers/lab instructors, curriculum development/learning 
specialists, tenure-track faculty with education specialty. 2.6 2.4 2.9 


Institutional support for electronic resources: 
e.g., journal subscriptions and databases. 3.9 3.9 4.0 


NOTE: PULSE Vision & Change Rubric 1.0. Each response scored as 0 (not observed), 1 (initial stages), 2 (average), 3 (very good), 4 (excellent, exemplary). Information was 
collected after the first year of the three-year demonstration (2014) and at the end of the three years (2016). The sample in 2014 included five institutional averages and 9 
departments at the other three schools (3 per school) that were averaged to make them comparable. The sample in 2016 included 33 individual departments across the eight 
universities, as well as scores provided by the Provosts at each institution, which were requested to contrast the view of a senior-level administrator with those of departments. 
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Learning Spaces 


UA has devoted funding to redesigning classrooms into collaborative learn- 
ing spaces (CLSs), including new furniture, additional projection equipment 
and other technology, carpeting, paint, and facilities work (including elec- 
trical upgrades, mounting projectors, etc.). Senior administrators have com- 
mitted to three additional years of funding up to $1 million per year for 
new CLSs and other classroom and lecture hall improvements. These new 
spaces are needed to accommodate the desire for large collaborative class- 
rooms (rooms for 130-250 student seats), since the institution has already 
targeted most of the large flat classrooms that can be converted to a CLS. 
The most popular CLSs are the larger rooms (90 to 264 seats). The university 
currently has ten CLSs ranging in size from 30 to 264 student-seat spaces, 
and its funded plan for summer, 2017 calls for ten additional CLSs ranging 
in size from 25 to 135. 


The next new building on UA's campus is likely to be an engineering build- 
ing, and the Dean of Engineering wants to include space for large collabo- 
rative classrooms. Administrators are also working to ensure that renovation 
of the Old Chemistry Building will include CLSs. Several department heads 
in STEM fields, including in chemistry and biochemistry, molecular and cel- 
lular biology, and several engineering departments, have become involved 
in space redesign with two associate heads on the leadership team. Several 
others have themselves taught in CLSs. For example, the department head 
of Chemical and Environmental Engineering co-taught with an engineering 
faculty member in a CLS; he supports team teaching in this facility. 


UPenn created several active-learning classrooms to accommodate growing 
demand. At the start of the AAU project and the SAIL program, UPenn had 
only one 42-seat active learning room since then, they have added six more 
active classrooms, including three that accommodate between 72 and 90 
students. All are fully scheduled throughout the year. One more collabora- 
tive classroom is planned, and faculty committees have asked the university 
to investigate further additions. The demand for and use of active learning 
rooms continues to grow, prompting the development of an additional active 
learning room in the Biomedical Library. 


UNC-Chapel Hill is experimenting with new classroom furniture and config- 
urations designed to make it easier for faculty members to use interactive 
learning methods. These classrooms facilitate eye contact between students, 
instructor access to students, and transitions between lecture, class discus- 
sion, and small group work. A series of classroom renovations are in prog- 
ress across the campus to support active learning efforts. Since 2010, UNC 
Chapel Hill has created sixteen active learning classrooms for general-pur- 
pose use. Early work featured small and mid-size classrooms including two 
smaller (45 seat) and one larger (90 seat) studio classrooms (larger round 
tables for student interaction). In 2015, the university’s first interactive lecture 
hall (a 100-seat classroom renovation for active learning) was completed. 
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Others are in progress, including a large SCALE-UP classroom, indicating a 
significant institutional trend toward support of active learning.’ To date, 
four primary designs have been used for active learning classrooms.”® 


At CU Boulder, the Provost’s Learning Spaces Committee in 2016 developed 
design principles for educational spaces that were informed by the AAU Initiative. 
These guidelines have been inserted in the “stage-gate” design process, before 
plans are made, as the concept for new space is considered by campus. 


Although redesigned learning spaces are not required for all forms of active 
learning, it can affect some applications directly. UC Davis, as an example, 
noted that implementation of active biological modeling in an introductory cal- 
culus course was highly negatively impacted by poorly designed instructional 
space. Discussion rooms that did not have tables where students could work 
together made group work and modeling exercises via computers logistically 
difficult and made it more difficult for students to complete the assignments. 


Learning Assistants 


Institutions have utilized both graduate and undergraduate students as 
teaching and learning assistants. Inclusion of students in instructional roles 
has benefits for institutions at the level of the course or section. With more 
trained individuals in the room, the capacity to facilitate and evaluate active 
learning activities increases. Including students also benefits the students 
themselves. Not only must they master the course material and develop 
familiarity with the objectives of various evidence-based learning techniques, 
but they also must put into play skills including leadership, evaluation and 
analysis, facilitation, and public speaking. Undergraduate learning assistants 
can play different roles in the classroom, with different levels of formality. 


Across the eight project sites, use of graduate and undergraduate assistants 
in active learning classes more than doubled, from 740 to 1,676, during the 
three years of the AAU project. 


Mevolaaliale wa wssisite] ales 


STUDENTS STUDENTS 
Year! 479 261 0 740 
Year2 649 278 4 931 
Year3 939 629 8 1676 
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WashuU utilizes Peer-Led Team Learning (PLTL) in its general chemistry and cal- 
culus classes, and is translating the model for engineering courses. PLTL groups 
typically consist of 6-8 students who work together to solve problems, and 
are facilitated by a Peer Leader. Peer Leaders are undergraduate students who 
have previous taken and performed well in the course.’ 


WashU’s model of PLTL is designed to help students become conscious of 
the problem-solving process. It also helps students develop important col- 
laboration skills, including how to approach problems effectively as a group, 
how to communicate well, and how to exchange and critique ideas in a 
collaborative environment. Participation in PLTL is voluntary, and students 
who do participate commit, through a contract, to meeting specific obli- 
gations to their group. Peer Leader training is an important component of 
the program. All Peer Leaders enroll in two one-credit courses. In these two 
courses, Peer Leaders learn how to be strong mentors for their groups, and 
they form a collaborative group of their own to help one another address 
common PLTL challenges. 


At UPenn, undergraduate learning assistants have been utilized in a similar 
way. For example, in introductory biology, two instructors restructured their 
large lecture into a hybrid active learning course with the help of undergrad- 
uate learning assistants. The hybrid biology course features traditional lec 
ture two days a week, and a problem-based active-learning session once a 
week where students apply concepts or are introduced to new topics. The 
success of this format encouraged these instructors to further adapt the 
course; they plan to integrate shorter active moments into the lecture por- 
tions of the course, and formalize student group assignments to encourage 
more productive group work. 


UA utilizes trained undergraduate learning assistants to help encourage stu- 
dent discussion (without giving the answer) and to help the instructor with 
formative assessment of student learning. At UA, as at several other institu- 
tions, undergraduate learning assistants are often students who have pre- 
viously taken and done well in the course. They are given credit for acting 
as learning assistants and further develop their leadership skills, but they do 
not need to be paid. 


One example of how UA uses undergraduates as learning assistants involves 
a measurement instrument called Fine-grained Evaluation of Active Learning 
(FEAL), which is designed for large classes in collaborative learning spac- 
es.8° FEAL is an activity quality measurement instrument that can be quickly 
administered by undergraduate learning assistants to record key measures of 
activity success such as student engagement, student success, activity diffi- 
culty, activity time, and associated lecture time. The instrument is designed to 


79 PLTL at WashU 

80 Sixing Ly, Loukas Lazos and Roman Lysecky, FEAL: Fine-Grained Evaluation of Active Learning in 
Collaborative Learning, paper presented at the American Society for Engineering Education Annual 
Conference & Exposition (June 25-28, 2017). 
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require minimal training and minimal effort within the classroom for recording 
observations. Quick administration of the instrument is critical because the 
learning assistants recording with FEAL are primarily tasked with engaging the 
students with the given activities. A key difference between FEAL and other 
tools (such as COPUS) is that the learning assistants must have expertise on 
the subject matter, as the intent is to evaluate activity quality. Moreover, rel- 
evant information such as the concepts covered by the activity are recorded 
and analyzed. The instrument is further applied to code exam questions and 
is used to correlate student performance for the same concepts. 


MSU is increasing the number of graduate and undergraduate students who 
assist in teaching active learning classes. Funds for increased numbers of grad- 
uate and undergraduate learning assistants in the main introductory biology 
courses (cell and molecular biology, and organismal and population biology) 
have been made permanent. These additional assistants provide the resources 
necessary to facilitate more student-centered approaches in class meetings, as 
well as to implement more frequent assessments and feedback, especially on 
constructed-response tasks. In chemistry, resources for undergraduate learn- 
ing assistants to support interactive engagement and the use of some con- 
structed response assessment items in the lecture sections of general chem- 
istry have been written into the departmental budget. This will be extended 
to organic chemistry in Fall 2017. 


At UNC-Chapel Hill, the computer science department began utilizing under- 
graduate learning assistants to provide academic support to their peers in lec 
tures, labs, office hours, and recitations, providing more individualized sup- 
port to students while strengthening their own understanding of the course 
material in the process. According to the department's website: “Within two 
years, the learning assistants became the linchpin of the department's intro- 
ductory courses.” The department has introduced the Learning Assistant 
Award, which recognizes an undergraduate learning assistant who has shown 
outstanding dedication to helping their fellow students understand and mas- 
ter challenging course material.®' 


Department Support Structures 


Another key to successful institutionalization of undergraduate instructional 
reforms is to align relevant administrative units, such as Centers for Teaching 
and Learning, with department-based instructional improvement efforts. 
For some institutions, this approach included linking advising and co-cur- 
ricular units with classroom instruction. For others, it meant consolidat- 
ing units to form a more effective organization. Providing college or cam- 
pus-wide structures to support departmental reform efforts increased the 
likelihood of institutionalization in AAU project sites. 
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Structural innovations offer one type of solution. One alternative is to embed 
pedagogical expertise in each department by hiring, for example, LPSOEs 
at UC Davis. Another is to develop a teaching-oriented center staffed by 
respected senior tenure-track faculty members engaged in STEM reform 
such as MSU's CREATE for STEM Institute and WashU’s CIRCLE. Brown 
provides another example, shifting the locus of control for STEM reforms 
from its Sheridan Center for Teaching and Learning (which remains highly 
engaged) to its academic departments. 


More culturally-oriented reforms also can assist in enhancing the value 
placed on instructional improvement. CU Boulder uses DATs to move away 
from appeals to individual faculty members and toward group consensus for 
reform. UA's FLCs have been fundamental in promoting improved pedagogy 
and in institutionalizing reforms. This cross-department approach is especially 
important when many departments lack sufficient numbers of faculty mem- 
bers interested or trained in improving instruction and reforming curricula. 


The AAU instructor survey also asked about use of on-campus and off-cam- 
pus professional development opportunities focused on improving teaching. 
These percentages increased between the two survey administrations, show- 
casing instructors’ additional interest in improving their teaching (see charts 
on page 89). 
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2014 2016 


On-Campus Professional Development Opportunity Users (%) Users (%) 


Teaching development events (i.e. talks, workshops) 
specifically for instructors. 40.7% 55.7% 


Teaching development opportunities and resources 


for NEW instructors. 23.0% 27.1% 
Peer evaluations/feedback of teaching. 41.6% 48.9% 
A mentor or other person to go to for advice about teaching. 45.2% 54.0% 


A center or unit focused on teaching and learning 
within your college or school. 21.7% 29.9% 


A university—wide center or unit focused on 
teaching and learning. 26.8% 37.7% 


University resources for instructors to improve 
their teaching methods 27.7% 43.3% 
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2014 2016 


Off-Campus Professional Development Opportunity —_ Users (% Users (%) 


Teaching development events (i.e. talks, workshops) 
specifically for instructors. 14.0% 20.6% 


Teaching development opportunities and resources 


for NEW instructors. 4.4% 7.4% 
A mentor or other person to go to for advice about teaching. 14.8% 20.6% 
A cohort of scholars focused on teaching and learning. 11.7% 20.7% 
Resources for instructors to improve their teaching methods. 18.6% 28.2% 
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Center tor Integrated Research In 
Cognition, Learning and Education 


The mission of the Center for Integrative Research on Cognition, Learning, 

and Education (CIRCLE) at WashU is to provide a bridge between STEM 

faculty and researchers in the cognitive and learning sciences to facili- 

tate collaborative projects to improve student learning. The Center con- 

nects the cognitive and learning sciences with classroom practices by: 

= fostering implementation of innovations in teaching across the 
university 

= supporting research to evaluate the effectiveness of these innovations 
in enhancing student learning and retention of knowledge, and 

= disseminating the results of these classroom-based evaluations using 
experimental methods to the university community as well as the to 
the national and international teaching and learning communities. 


CIRCLE was founded in 2011. It is funded by the Office of the Provost, 
though its leadership consists of STEM faculty members, which gives 
it particular cachet with departmental faculty and staff, with whom it 
operates in a partner role than rather than as a support structure. CIRCLE 
serves as a coordinating body for many STEM education-related activi- 
ties on campus, including faculty learning communities and the Faculty 


Fellow program. The Center works on efforts that are part of the AAU 
Initiative, as well as those that fall outside the project. 


The center recently obtained a four-year commitment to support its new 
Transformation for innovation in STEMS (TIES), which is adapted from 
Carl Wieman’s University of Colorado Boulder Science Education Initiative. 
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CREATE for STEM is a Michigan State University sponsored research 
institute with a broad mandate for Collaborative Research in Education, 
Assessment and Teaching Environments for the STEM fields. The Institute 
is a collaboration between the colleges of Education, Natural Science, 
Engineering, and the Lyman-Briggs Residential College in coordination 
with the Office of the Provost. 


CREATE for STEM was implemented and endowed with university funds 
to coordinate educational activities across campus. (These activities in- 
clude those associated with the AAU project as well as others not as- 
sociated with the project.) The Institute brings together STEM faculty 
and faculty from the College of Education and serves as a springboard 
and hub for innovation, research, and intellectual collaboration to help 
bring to fruition ideas and projects that will make a difference in the 
teaching and learning of STEM for all learners. The Institute’s work has 
five major focus areas: 
1. creating and investigating change in K-16 STEM education, 

educational policy in STEM fields, 

discipline-based undergraduate STEM education, 

developing innovative and digital materials, and 

international engagement with the global STEM education 

community. 


Within each of these areas, the Institute builds interdisciplinary and inter- 
generational research groups, fosters new talent, provides seed money for 
initial work, and supports the grant writing process. CREATE for STEM is 
also a hub for the exchange of information and ideas, sponsoring confer- 
ences, workshops, seminars, and visiting scholars to enhance the interest 
in STEM education and develop capacity in these fields here at MSU. 


One key feature of the Institute that distinguishes it from many other 
teaching and learning centers is that rather than being staffed adminis- 
trators, it houses respected tenured and tenure-track faculty members 
engaged in STEM reform. This provides a different relationship between 
the Institute and departments. 
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Cultural Change 


Department Ownership of the Curriculum 


One key to improving undergraduate STEM education is for a department and its 
faculty members to recognize and acknowledge their collective responsibility to 
improve the effectiveness of introductory/foundational courses. In a research uni- 
versity achieving such a departmental commitment is challenging both because 
many (sometimes most) of the students taking these introductory courses will 
not major in the department offering them and because the attractions and 
rewards of research universities for faculty members most often are in research 
and graduate education. 


AAU has observed that departments most likely to emphasize evidence-based 
active-learning strategies in foundational courses have thought deeply about the 
curricula and content of these courses, along with ways to assess student learn- 
ing in them. Ultimately, collective responsibility for shared learning objectives by 
course will necessitate developing a uniform vision of educational improvement 
among faculty members within and across departments, as well as the develop- 
ment of mechanisms to assess progress in teaching effectiveness for all students. 


Consider the Biology Initiative at MSU as an example of this process. The 
Biology Initiative focused on introductory genetics, which serves three colleges 
and 18 majors. No single college or department “owned” the course, which 
made it difficult to assign teachers, develop common learning objectives, and 
assess results. Funded by the Provost and coordinated with the AAU project 
at MSU, the first step in this reform initiative was to create a management 
committee. The committee, which reported to the Dean of Natural Sciences, 
was responsible for introductory genetics, including the assignment of instruc- 
tors and the development a set of common learning objectives. This arrange- 
ment increased cross-department communication and led to more coordi- 
nated instructional and assessment practices. Follow-up assessment showed 
a substantial increase in student performance relative to the prior format of 
introductory genetics. 


As another example, at UC Davis, the Colleges of Bioscience, Agriculture, and 
Environmental Science, as well as the Department of Chemistry, recently devel- 
oped a shared five-course sequence for students in the Life Sciences (piloted 
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in Fall 2017 with the initial work on three quarters). This collaboration rep- 
resents the first time that such a diverse group of faculty members and depart- 
ments cooperated to develop new courses and a course sequence. UC Davis 
then linked its summer bridge program with the new curricular sequence and 
identified discrete paths for students to take chemistry courses based on their 
interests and intended majors. 


Since the start of the AAU project, UNC-Chapel Hill has transformed all of its 
introductory courses in Biological Sciences in a manner consistent with evi- 
dence-based pedagogies. Physics as a department opted to reform all intro- 
ductory courses in a similar manner. One key: Regardless of instructor there 
will be common materials and examinations in these introductory courses. 


Among the most visible evidence of departmental ownership of courses and 
curricular is at Brown. The project initially was housed in the Sheridan Center 
for Teaching and Learning with the Director playing a lead role in the project. 
When the Director unexpectedly left, the chairs in key STEM departments 
took responsibility for the project and made sure that the departments took 
responsibility for course and curriculum improvements. 


Institutional Commitment to Long-term STEM Reforms 


Institutionalizing reforms of undergraduate STEM education at research univer- 
sities eventually requires internal institutional investment of resources; it cannot 
be achieved solely by a series of externally-funded grants. The nature of this 
investment can vary, supporting personnel, infrastructure, and/or space. Public 
pronouncements of support by university leaders are also important contribu- 
tions to the spread of instructional reforms across departments. 


Each of the eight project sites has garnered institutional support for long-term 
STEM undergraduate reforms. At Brown, the Office of the Provost has contin- 
ued its pledge of budgetary support for the project despite turnover in that 
position. The Provost and dean have committed institutional resources to assist 
participating departments in reforming introductory STEM courses (including 
hiring and training Teaching Assistants). In addition to financial resources, the 
central administration has made public pronouncements in support of the AAU 
STEM reforms and has engaged with project activities. Brown also has modified 
some teaching spaces to better reflect the use of active learning techniques. 
The most significant long-term commitment, however, is signified by the hir- 
ing of a specialist, working in the Sheridan Center for Teaching and Learning, 
who will help train problem-solving group facilitators. This move by the uni- 
versity is part of an emerging project to sustain and strengthen the culture of 
problem-solving created by the AAU project. The project is a large-scale, joint 
effort between the Sheridan Center, the Office of the Dean of the College, and 
the STEM department chairs and faculty. 


At MSU, the Provost, deans, and chairs have made efforts to encourage aca- 
demic programs to hire individuals knowledgeable about evidence-based 
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teaching. Chemistry hired an individual whose sole job focus is to support 
curriculum transformation. Also in Chemistry, the Provost provided funds 
for undergraduate learning assistants (ULAs) and is funding the alteration 
of classroom space. The Provost funded new graduate and undergraduate 
teaching assistants for introductory biology courses. Three new course cur- 
riculum coordinators were hired with university funds; their focus is on the 
introductory sequence in biology. Physics hired several ULAs for teaching 
introductory courses. 


At UPenn, one key to increasing the number of faculty involved and institu- 
tional support has been the evidence of success—students taking reformed 
courses achieve better than those taking traditional courses and seem to per- 
sist in STEM as well (the evidence of this outperformance is especially strong in 
bioengineering and physics). The evidence of improved student performance 
has led faculty and chairs to vote to sustain the SAIL classroom reform model. 
UPenn has also received central administrative support for building new class- 
rooms that fit active learning instructional strategies. 


Despite substantial institutional budgetary constraints, UA received a pledge 
to continue funding of the core faculty learning community and workshop 
efforts. UA leaders have also contributed resources 
to reform classrooms and provide salary for some 
teaching-related positions. Central administrators 
also have regularly reported on the AAU project 
achievements in campus news outlets. UA com- 
mitment of resources to long-term reforms include 
funds for a new Center for University Educational 
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Scholarship (including a director and three fellows). 


Extensive turnover by senior university leadership 
at CU Boulder in recent years, makes it difficult 
to assess the stability of long-term commitments 
to reform by central administration. However, CU 
Boulder received institutional funds for three new 
STEM teaching-oriented faculty hires and the central 
administration now requires departments to answer 
three questions about its effectiveness. This signals 
that the central administration is cooperating with 
AAU project goals. 


The central administration and various colleges at 
UC Davis now contribute funds to support the STEM 
reforms started by the AAU project. For example, 
the Colleges of Biology and Agriculture now sup- 
port the continued reform of Chemistry Ill. The 
Provost now commits money for the CEE, multi- 
ple faculty learning communities, an evidence-based 
course redesign institute, and course design awards. 


Questions 


How does your department 
define excellence with respect to 
the activities that it carries out 
and what values and behaviors 
are associated with excellence? 


How do the efforts within the 
department lead to the inclu- 
sion of all students in achieving 
academic success, student social 
development, and enhancing 
the reputation of the depart- 
ment and the university? 


Given the department’s defini- 

tion of excellence and inclusion, 
how will the department attain 
inclusive excellence? 
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UNC-Chapel Hill provides an example of the investment of resources at 
the precise time needed to expand reforms: The Provost and several Deans 
agreed to contribute university funding to continue the course buy-out strat- 
egy (i.e., both mentors and apprentices get a full course credit for co-teach- 
ing one course) after AAU funds disappeared, which enabled UNC-Chapel 
Hill to expand the numbers of participants in its STEM reform programs. 
Beyond its support for the original AAU grant, UNC-Chapel Hill central 
administration now supports salaries for five STEM instructors experienced 
in Discipline Based Education Research (DBER) and evidence-based teaching, 
the Chancellors Scholars program (for minority students), summer teaching 
institutes for faculty members, and classroom renovations. In addition, the 
Quality Enhancement Program (QEP) at UNC-Chapel Hill has been funded 
until at least 2022. The QEP will take over the work started by the AAU proj- 
ect after funding ends. 


The substantial resource commitment to CIRCLE is indicative of an institu- 
tional commitment to the overall reform of STEM education at WashU. CIRCLE 
recently obtained a four-year commitment to support its new Transformational 
Initiative for Education in STEM (TIES), which is adapted from Carl Wieman’s 
Science Education Initiative. The Chemistry Department has also funded a 
permanent position to disseminate and institutionalize educational reforms 
in that department. 


Leveraging Other Undergraduate Reforms on Campus 


Project sites successfully leveraged existing undergraduate reform efforts in 
their efforts to broaden the reform of STEM education. Taking advantage of 
resources already committed to other types of educational reforms, either 
from external grants or from internal sources, was a common theme. MSU 
was able to leverage various ongoing projects to create a larger, more visible, 
and longer-lasting presence on campus. 


Better coordination of various administrative units responsible for undergrad- 
uate education also was evident. Successful forms of coordination included 
linking advising and co-curricular units with classroom instruction as well as 
consolidating units to form a more effective organization. UC Davis is a prime 
example of the latter. UC Davis has given greater visibility to educational 
reform efforts on campus by merging the iAMSTEM Center with the Center for 
Excellence in Teaching and Learning (CETL) to form the Center for Educational 
Effectiveness (CEE) and providing funds for CEE staff.®* CEE is now supported 
with regular university funds and works with Student Affairs Administration 
(SAA) to combine non-curricular information about students with classroom 
data to better identify factors in student success. 


Linking STEM educational reforms with university initiatives also proved suc 
cessful. As a prime example, at UNC-Chapel Hill, the Chancellor's Strategic 
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Framework now requires all 13 schools and their departments to show evidence 
of high impact research and innovative teaching. The Quality Enhancement 
Program (QEP) promoted by central administration includes a focus on improv- 
ing undergraduate STEM education. The UNC-Chapel Hill project also has 
emphasized the improvement of academic performance and degree retention 
among underrepresented minorities, which is a university priority. 


Demonstrating more effective ways to measure the quality of teaching also 
has been used to support institutional and state-wide efforts to improve 
teaching. As an example, at UA, the AAU project has taken advantage of 
university resources for improving teaching and learning, incorporating activ- 
ities and staff into an overall STEM reform movement. Two additional NSF 
grants have supported an expansion of AAU project-initiated efforts. Finally, 
the UA team has cited state-level policy emphasizing the quality of educa- 
tion on public campuses to promote STEM reforms at UA. 


All project sites were able to leverage the reputation of AAU in extending 
the visibility of their projects. Also, membership in other national networks 
enabled project sites to leverage external support for local reforms.® 


Institutional Commitment to Align Faculty Rewards 
to Evidence-based Teaching Practices 


A primary goal of this initiative was to bring about a shift in the culture 
of research universities to increase the use and valuing of evidence-based 
instruction to the point where it became the norm rather than the province 
of a dedicated few. AAU was aware that mainstreaming evidence-based 
pedagogy would require aligning the faculty reward structure with (often) 
new expectations for teaching, realigning rewards to reinforce an expec- 
tation for teaching excellence consistent with the use of evidence-based 
instruction. Of all the project goals, changing faculty rewards to increase 
the value of teaching has been the most difficult. Despite AAU’s expecta- 
tion as expressed in the proposal process, only two of the eight project sites 
proposed actual plans to work on the routines by which their campus nor- 
mally addressed merit, promotion, and tenure judgments, including taking 
this up with the political entities, like faculty senates, that would have to 
be on board for widespread change to occur. 


In spite of this lack of emphasis by projects on and the apparent resis- 
tance to systematically address this aspect of culture change, AAU saw 
clear trends over the years toward aligning the institutional incentive struc- 
ture with support of evidence-based teaching. For example, Physics at MSU 
now requires every pre-tenured faculty member to have at least one for- 
mal observation by a trained professional (in evidenced-based teaching) 
each semester. Tenured faculty members must do so at least once per year. 
These findings are now an official part of the faculty dossier in Physics. CU 


83 CIRTL Network 


Undergraduate STEM Education Initiative 


97 


98 


Boulder reformed its institutional requirements for departmental and faculty 
performance to include excellence in teaching. At WashU, for promotion 
and tenure, the Schools of Arts and Science and Engineering and Applied 
Science now require faculty members submitting materials for performance 
review to include details about their teaching and participation in profes- 
sional development activities (teaching certificates from CIRCLE are help- 
ful in this regard). In the past only student course ratings were required for 
faculty review. At UA the promotion and tenure process at the university 
level has been modified to include a required teaching portfolio and peer 
assessment/observation of classroom instruction. In the future as part of its 
regional accreditation and monitoring by The University of Arizona higher 
learning commission, UA will gather data on the use of evidence-based 
teaching and the quality of instruction. UNC-Chapel Hill similarly will gather 
data on teaching and learning as part of its regional accreditation. 


AAU used two data sets for drawing more general inferences about the 
place of evaluation and assessment of teaching in judging faculty for 
merit, promotion, and tenure. As noted above, information about this was 
requested in the annual reports. Second, in each of the two rounds of 
common data collection across the sites (2014 and 2016), this issue was 
addressed in two ways. The survey included questions that elicited faculty 
perception of how this work is valued by their department and institution. 


= Perceptions of recognition of importance of teaching by departmental and 
campus administrators (>3.0) out of sync with perceptions of the role effec 


tive teaching plays in annual review and salary (=2.5). 


= Most felt quality of evidence for teaching used was of low (about 33%) or 
medium (about 50%). Only about 15% judged the quality high. 


How would you rate the quality of evidence for teaching used 
in the following circumstances? 


2014 By your College in the annual review and salary process 


2016 By your College in the annual review and salary process 


2014 By your College in the annual review and salary process h_—lC(<tkt:lwUltéCUd 


2016 By your College in the annual review and salary process 


(=) 


Jas 50 Te 100 
Blow @Medium MHigh 
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Teaching Quality Framework 


One of the barriers to including information about teaching in tenure and 
promotion decisions is the lack of a scholarly approach to teaching evalua- 
tion. The University of Colorado Boulder is developing a Teaching Quality 
Framework (TQF) to address this shortcoming and provide a common 
campus-wide framework for evaluating teaching quality that is contex- 
tualized to and enacted within varying disciplines, units, and roles. The 
TQF effort will empower and support departments to define the multiple 
measures that specify teaching quality in a way that is evidence-based, 
draws from leading scholarship on educational practices and evaluations, 
and is relevant to individual domains. The goal of the framework is both 
to provide better mechanisms for assessing teaching quality for merit, 
tenure, and promotion and to support improved teaching by providing 
faculty members with feedback that they can use to improve as educators. 


The TQF defines a professional approach to teaching as including: clear 
goals, adequate preparation, appropriate methods, significant results, ef- 
fective presentation, and reflective critique. These form the assessment 
categories for the framework, which are the same across all departments. 
However, interpretation of these categories and their relative weights 
are defined at the departmental level. Within each category, quality of 
teaching is assessed using three standard sources: the faculty member, 


students, and peers. 


Implementing the TQF follows an opt-in model, with pilot departments 
choosing to engage and become leaders in this process. Two layers of 
action advance this initiative. First, departmentally based TQF Teams con- 
sist of three to six participants per department, who meet regularly and 
approach their work using a Departmental Action Team model (see sep- 
arate call-out box). These teams contextualize the elements of the frame- 
work in their discipline and decide what is required for implementation in 
their department. Second, a campus-wide TQF Taskforce includes a broad 
set of participants (departmental representatives, deans, and other key 
stakeholders) who incorporate changes into the overall framework and 
communicate among departmental teams and other stakeholders. Key 
to the program’s success is a facilitator of the multiple departmental TQF 
teams who also serves as a communication channel between those teams 
and the campus taskforce. 


To date, nine departments are committed to the process. These depart- 
ments include humanities, social sciences, natural sciences, and engi- 
neering. An additional four departments have expressed interest. A cam- 
pus-wide community has met each semester to discuss the TQF process 
and the resources to support its continuing implementation. 
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The heads of participating departments were asked to write brief statements 
about how teaching is evaluated and valued in their department. In 2014, 30 
department chairs from seven of eight institutions responded. In 2016, 30 
department chairs from eight institutions responded. In both years, respon- 
dents asserted that teaching is highly valued. Many of them quoted university 
level or college level language to this effect. Most provided some level of detail 
about their process for collecting information and providing feedback to faculty. 


Student evaluations at the end of courses were overwhelmingly the metric 
used in 2014. Faculty personal statements, review of course materials, and 
observation also played some role, especially in tenure decisions. It was not 
possible to tell from most of the department head statements whether these 
additional criteria made any explicit address to use of evidence-based instruc 
tional methods. In 2016, student evaluations were still the major metric, but 
there were two statements that could be taken as critiques of their useful- 
ness. One of these is mild — a chemistry department notes that “scores will 
be considered, but not weighted heavily unless they are extraordinarily pos- 
itive or negative.” The other is more robust — a mathematics department is 
engaged in a debate over whether to use them at all, given the literature on 
implicit bias and other confounding elements in their use. 


By 2016 most departments mention some level of peer observation, but in only 
a few cases was it clear that observers would use tools that helped them go 
beyond their own experience. Many departments expect that tenure review 
will include a portfolio that contains representative course materials. 


In 2016, twelve of the 30 made explicit reference to evidence-based pedagogy 
as an important criterion for tenure track faculty, and two more included it for 
non-tenure track faculty but not tenure track, as compared with six and two 
respectively out of 30 in 2014. Eight had statements classified as permissive, 
that is, allowing faculty to present use of evidence-based work without implying 
it is expected, compared to six in 2014. Binning the ranges shows that atten- 
tion to this was evident in 73% of the responses in 2016, compared to 47% 
in 2014. Three among the ones noted above for 2016 specifically mentioned 
student outcomes as a criterion: one provided no detail, another tracks stu- 
dent success in later courses, and the third tracks assessment of achievement 
of learning goals that have been agreed by the department. 


Between 2014 and 2016 there was a shift in the tone of the responses that, 
taken collectively, points to culture change, at the very least at the level of chairs 
being more willing to think beyond the cookbook answers to the presenting 
question. For example, there was more of a tendency for the chair to say or 
imply that keeping a focus on good teaching is an important part of the chair’s 
role. The statements from one institution (five departments) all emphasized, 
and took pride in, their use of the resources of their Teaching Center. Finally, 
some chairs took the question as an invitation to comment on culture change 
more generally and spoke of seminar presentations and an increase in depart- 
ment conversations about teaching, as well as named awards for teaching. 
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There were cases in which an aspect of the site’s AAU project plan was partic 
ularly evident in the response. These include CU Boulder’s beginning to intro- 
duce very explicit expectations of an effort to use evidence-based pedagogy 
in the merit review criteria; and UPenn Math Department's very robust reex- 
amination of the use of student ratings and alternative metrics, based on their 
experiments in introductory math. Another institution, on the other hand, 
presented a striking difference between the focus of the project leadership, 
which has tended to espouse the view that faculty motivation to change is 
intrinsic and not driven by the institutional reward system, and the responses 
from its chairs which also cite its Natural Sciences college-wide resources. The 
College policy is explicit in its criteria in a way that points to use of modern 
pedagogy as desirable. 


In general, the trends observed in the department heads surveys are echoed 
in the annual reports from project leadership, although not perfectly, as noted 
above. It is useful to note that the last five years have seen the appearance of 
significant new tools for peer observation and self-analysis of teaching prac 
tice, so that it is easier now for a department that seeks alignment of incen- 
tives with good practice to begin to articulate what is needed. Examples of new 
approaches are presented in this section and in section 3, which come not only 
from a funded project site but several members of the larger AAU network. 
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AAU STEM Project Site Scholarship 


This section lists the peer reviewed scholarship that has been generated by the 
AAU Project Site Teams. The citations are categorized by their area of focus: 
Policies and Strategies; Elements of Active Learning; Practical Steps to Improve 
Undergraduate Education; and Instructional Personnel. 


Policies and Strategies 


Institutional Strategies to Increase the Value of Improving Undergraduate Teaching 

Corbo, J. C., Reinholz, D. L., Dancy, M. H., Deetz, S., & Finkelstein, N. (2016). Framework for 
transforming departmental culture to support educational innovation. Physical Review Physics 
Education Research, 12(1), 010113. http://dx.doi.org/10.1103/PhysRevPhysEducRes.12.010113 

Corbo, J. C., Reinholz, D.L., Dancy, M. H. and Finkelstein, N (2015). Departmental Action Teams: 
Empowering Faculty to Make Sustainable Change, in Physics Education Research Conference 
2015 (College Park, MD, 2015). http://dx.doi.org/10.1119/perc.2015.pr.018 

Fairweather, J. S., Trapani, J., & Paulson, K. (2016). The roles of data in promoting institu- 
tional commitment to undergraduate STEM reform: The AAU STEM Initiative experience. 
In G. C. Weaver, W. D. Burgess, A. L. Childress, & L. Slakey (Eds.), Transforming institu- 
tions: Undergraduate STEM education for the 21st century (pp. 429-437). West Lafayette, 
IN: Purdue University Press. 

Macfadyen, L. P., Groth, D., Rehrey, G., Shepard, L., Greer, J., Ward, D., Bennett, C., Kaupp, 
J., Molinaro, M., & Steinwachs, M. (2017). Developing institutional learning analytics ‘com- 
munities of transformation’ to support student success. Paper presented at the Seventh 
International Learning Analytics & Knowledge Conference, March 13-17, Vancouver, 
British Columbia, Canada. Retrieved from http://bayviewalliance.org/bvwwp/wp-content/ 
uploads/2016/11/LA Fellows workshop V3.0.pdf 

Matz, R. L., & Jardeleza, S. E. (2016). Examining the Role of Leadership in an Undergraduate 
Biology Institutional Reform Initiative. CBE-Life Sciences Education, 15(4) 

Miller, E. R., & Fairweather, J. S. (2016). The role of cultural change in large scale STEM reform. 
In G. C. Weaver, W. D. Burgess, A. L. Childress, & L. Slakey (Eds.), Transforming institu- 
tions: Undergraduate STEM education for the 21stcentury (pp. 48-66). West Lafayette, IN: 
Purdue University Press. 

Pagliarulo, C., & Molinaro, M. (2015). iAMSTEM: Increasing STEM success at UC-Davis. Peer 
Review, 17(2), 13. Retrieved from https:/Awww.aacu.org/peerreview/2015/spring/pagliarulo. 

Weaver, G. C., Burgess, W. D., Childress, A. L., Slakey, L. (Eds.). (2016). Transforming Institutions: 
Undergraduate STEM education for the 21st century. West Lafayette, IN: Purdue University 
Press. http:/Awww.thepress.ourdue.edu/titles/format/9781557537249 


Policies and Strategies to Encourage the Use of Effective Teaching Practices 

Burd, G. D., Tomanek, D., Blowers, P., Bolger, M., Cox, J., Elfring, L., Grubbs, E., Hunter, J., 
Johns, K., Lazos, L., Leysecky, R., Milsom, J. A., Novodvorsky, |., Pollard, J., Prather, E., 
Talanquer, V., Thamvichai, K., Tharp, H., & Wallace, C. (2016). Developing faculty cultures 
for evidence-based teaching practices in STEM: A progress report. In G. C. Weaver, W. D. 
Burgess, A. L. Childress, & L. Slakey (Eds.), Transforming institutions: Undergraduate STEM 
education for the 21st century (pp. 90-102). West Lafayette, IN: Purdue University Press. 
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Hogan, K. A., Krumper, J., McNeil, L. E., & Crimmins, M. T. (2016). Advancing evidence-based 
teaching in gateway science courses through a mentor-apprentice model. In G. C. Weaver, 
W. D. Burgess, A. L. Childress, & L. Slakey (Eds.), Transforming institutions: Undergraduate 
STEM education for the 21stcentury (pp. 77-89). West Lafayette, IN: Purdue University Press. 

Reinholz, D. L., Corbo, J. C., Dancy, M. H., Finkelstein, N., & Deetz, S. (2016). Towards a model 
of systematic change in university STEM education. In G. C. Weaver, W. D. Burgess, A. L. 
Childress, & L. Slakey (Eds.), Transforming institutions: Undergraduate STEM education for 
the 21stcentury (pp. 115-124). West Lafayette, IN: Purdue University Press. 


Practical Steps to Improve Undergraduate Education: Curricula and Assessment 


Assessment of Undergraduate Student Learning 

Becker, N., Noyes, K., & Cooper, M. (2016). Characterizing students’ mechanistic reasoning 
about London dispersion forces. Journal of Chemical Education, 93(10), 1713-1724. http:// 
dx.doi.org/10.102 1/acs.jchemed.6b00298 

Cooper, M. M. (2015). Why ask why? Journal of Chemical Education, 92(8), 1273-1279. http:// 
dx.doi.org/10.102 1/acs.jchemed.5b00203 

Cooper, M. M., Caballero, M. D., Ebert-May, D., Fata-Hartley, C. L., Jardeleza, S. E., Krajcik, 
J. S., Laverty, J. T., Matz, R. L., Posey, L. A., & Underwood, S. M. (2015). Challenge faculty 
to transform STEM learning. Science, 350(6258), 281-282. http://dx.doi.org/10.1126/sci- 
ence.aab0933 

Cooper, M. M., Kouyoumdjian, H., & Underwood, S. M. (2016). Investigating students’ rea- 
soning about acid—base reactions. Journal of Chemical Education, 93(10), 1703-1712. http:// 
dx.doi.org/10.102 1/acs.jchemed.6b00417 

Cooper, M. M., Williams, L. C., & Underwood, S. M. (2015). Student understanding of inter- 
molecular forces: a multimodal study. Journal of Chemical Education, 92(8), 1288-1298. 
http://dx.doi.org/10.102 1/acs.jchemed.5b00169 

Cooper, M. M. (2016). It is time to say what we mean. Journal of Chemical Education, 
93(5),799-800. http://dx.doi.org/10.102 1/acs.jchemed.6b00227 

Cooper, M. M., Posey, L. A., & Underwood, S. M. (2017). Core ideas and topics: Building up 
or drilling down? Journal of Chemical Education, 94(5), 541-548. http://dx.doi.org/10.1021/ 
acs.jchemed.6b00900. 

Irving, P. W., Sawtelle, V. D., & Caballero, M. D. (2015). Troubleshooting Formative Feedback 
in P3 (Projects and Practices in Physics). Paper presented at the Physics Education Research 
Conference, July 29-30, College Park, MD. http://dx.doi.org/10.1119/perc.2015.pr.034 

Laverty, J. T., Underwood, S. M., Matz, R. L., Posey, L. A., Carmel, J. H., Caballero, M. D., 
Fata-Hartley, C. L., Ebert-May, D., Jardeleza, S. E., & Cooper, M. M. (2016). Characterizing 
College Science Assessments: The Three-dimensional Learning Assessment Protocol. PLOS 
ONE, 11(9), e€0162333. http://dx.doi.org/10.1371/journal.pone.0162333 

Underwood, S. M., Reyes-Gastelum, D., & Cooper, M. M. (2015). Answering the Questions of 
Whether and When Learning Occurs: Using Discrete Time Survival Analysis to Investigate the 
Ways in Which College Chemistry Students’ Ideas About Structure—Property Relationships 
Evolve. Science Education, 99(6), 1055-1072. http://dx.doi.org/10.1002/sce.21183 

Williams, L. C., Underwood, S. M., Klymkowsky, M. W., & Cooper, M. M. (2015). Are 
Noncovalent Interactions an Achilles Heel in Chemistry Education? A Comparison of 
Instructional Approaches. Journal of Chemical Education, 92(12), 1979-1987. http://dx.doi. 
org/10.1021/acs.jchemed.5b00619 
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Course (Re-)Design and Student Learning Outcomes 

Crimmins, M. T., & Midkiff, B. (2017). High structure active learning pedagogy for the teach- 
ing of organic chemistry: Assessing the impact on academic outcomes. Journal of Chemical 
Education, 94(4), 429-438. http://dx.doi.org/10.1021/acs.jchemed.6b00663 

Dockter, D., Uvarov, C., Guzman-Alvarez, A., & Molinaro, M., (in press). Improving prepara- 
tion and persistence in undergraduate STEM: Why an online, summer chemistry prepara- 
tion course makes sense. Forthcoming ACS Publication, Online Approaches to Chemical 
Education, Manuscript ID: bk-2016-005344. 

Dourte Segan, L., & Elliott, E. R. (2016). Group problem solving coupled with hands-on activ- 
ities: Conceptual gains and student confidence in an introductory biomechanics course. 
Paper presented at the American Society for Engineering Education Annual Conference 
& Exposition, June 26-29, New Orleans, LA. Retrieved from httos://www.asee.org/public/ 
conferences/64/papers/16930/view 

Dourte Segan, L., & IIkhanipour Rooney, S. (2015). Structured, active, in-class learning: 
Connecting the physical to the mathematical in an introductory biomechanics course 
(work in progress). Paper presented at the American Society for Engineering Education 
Annual Conference & Exposition, June 14-17, Seattle, WA. Retrieved from https://peer. 
asee.org/24745 

Eddy, S. M., & Hogan, K. A. (2014). Getting under the hood: How and for whom does increas- 
ing course structure work? CBE—Life Sciences Education, 13(3), 453-468. http://dx.doi. 
org/10.1187/cbe.14-03-0050 

Irving, P. W., Obsniuk, M. J., & Caballero, M. D. (2016). P?: A Practice Focused Learning 
Environment. arXiv:1607.04455 [physics.ed-ph]. Retrieved from http://arxiv.org/ 
abs/1607.04455. 

Molinaro, M., Steinwachs, M., Li, Q., & Guzman-Alwarez, A. (2016). Promoting instructor 
and department action via simple, actionable tools and analyses. Paper presented at the 
Learning Analytics for Curriculum and Program Quality Improvement Workshop co-lo- 
cated with the Sixth International Learning Analytics and Knowledge Conference, April 25, 
Edinburgh, United Kingdom. Retrieved from: http://ariadne.cti.espol.edu.ec/pcla2016/files/ 
ProceedingsPCLA2016.pdf 

Rainey, K., Corbo, J. C., Reinholz, D. L., and Betterton, M. Improving Representation in Physical 
Sciences using a Departmental Action Team, in Physics Education Research Conference 2016 
(Sacramento, CA, 2016). http://dx.doi.org/10.1119/perc.2016.pr.061 

Solomon, E. D., Cohen, C. A., Cahill, M. J., McDaniel, M. A., Zhao, J., & Frey, R. F. (in review). 
Examining the impact of student characteristics: Active learning with clickers predicts per- 
formance in an introductory science curriculum. CBE-Life Sciences Education. Manuscript 
submitted May 2017. 

Underwood, S. M., Reyes-Gastelum, D., & Cooper, M. M. (2016). When Do Students Recognize 
Relationships Between Molecular Structure and Properties? A Longitudinal Comparison of 
the Impact of Traditional and Transformed Curricula. Chemistry Education Research and 
Practice, 17(2), 365-380. http://dx.doi.org/10.1039/C5RP00217F 


Laboratory Instruction in STEM 

Carmel, J. H., Ward, J. S., & Cooper, M. M. (2017). A Glowing Recommendation: A Project- 
Based Cooperative Laboratory Activity to Promote Use of the Scientific and Engineering 
Practices. Journal of Chemical Education, 94(5), 626-631. http://dx.doi.org/10.1021/acs. 
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Elements of Active Learning 


Collaborative Small-Group Learning 

Obsniuk, M. J., Irving, P. W., & Caballero, M. D. (2015). A Case Study: Novel Group Interactions 
through Introductory Computational Physics. Paper presented at the Physics Education 
Research Conference, July 29-30, College Park, MD. http://dx.doi.org/10.1119/perc.2015.pr.055 

Trenshaw, K. F., Targan, D. M., & Valles, J. M. (2016). Closing the achievement gap in STEM: 
A two-year reform effort at Brown University. Paper presented at the American Society for 
Engineering Education Northeast Section Conference, April 28-30, Kingston, Rhode Island. 
Retrieved from http://egr.uri.edu/wp-uploads/asee2016/73-1064-1-DR.pdf 


Creating Learning Resources 

Allen, G., Guzman-Alvarez, A., Smith, A., Gamage, A., Molinaro, M., & Larsen, D. S. (2015). 
Evaluating the effectiveness of the open-access ChemWiki resource as a replacement for 
traditional general chemistry textbooks. Chemistry Education Research and Practice, 16, 
939-948. http://dx.doi.org/10.1039/C5RP00084J) 


Instructional Personnel 


Training Graduate Teaching Assistants 

Becker, E. A., Easlon, E. J., Potter, S. C., Guzman-Alvarez, A., Spear, J. M., Facciotti, M. T., Igo, 
M. M., Singer, M., & Pagliarulo, C. (Manuscript submitted for publication). The effects of 
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Sciences Education. http://dx.doi.org/10.1101/115295 

IIkhanipour Rooney, S., Schafer McGurk, J., Elliott, E. R., Williams, U. J., & Dourte Segan, L. 
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in-class learning environment as a teaching assistant. Paper presented at the American 
Society for Engineering Education Annual Conference & Exposition, June 14-17, Seattle, 
WA. Retrieved from https://peer.asee.org/24092 


Evaluating Teaching Practices 

Burd, G. D., Novodvorsky, |., Tomanek, D., & Varma-Nelson, P. (2015). Improving teaching qual- 
ity through peer review of teaching. In Searching for Better Approaches: Effective Evaluation 
of Teaching and Learning in STEM (pp. 47-58). Tucson, AZ: Research Corporation for Science 
Advancement. Retrieved from http://rescorp.org/gdresources/publications/effectivebook.pdf 

Fisher, B. A., & Frey, R. F. (2015). Using documentary tools to foster the practice of scholarly 
teaching. National Teaching and Learning Forum, 24(2), 4-6. http://dx.doi.org/10.1002/ 
ntlf.30016 
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Pondugula, S., & Frey, R. F. (2016). A visual approach to helping instructors integrate, docu- 
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Solomon, E. D., Repice, M. D., Mutambuki, J. M., Leonard, D. A., Cohen, C. A., Luo, J., & Frey, 
R. F. (in review). A mixed method investigation of clicker implementation styles in STEM. 
CBE-Life Sciences Education. Manuscript submitted August 2017. 
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Since the start of its Undergraduate STEM 
Education Initiative, AAU's objective has 
been to include as many member universities 
as possible in activities that improve under- 
graduate STEM teaching and learning. 


The high levels of interest in the Initiative — for example, AAU received con- 
cept papers from 31 universities to become project sites, but had the ability 
to seed-fund only eight — revealed the potential for a great deal of action. 
To further leverage the national impact of the Initiative, AAU believed that all 
member institutions should have the opportunity to engage. Moreover, other 
national examples demonstrate the effectiveness of networks and learning 
communities in spreading innovative practices in STEM education. 


At the time of evaluating concept papers and selecting project sites, AAU 
took steps to encourage all member universities to continue to participate in 
the Initiative. First, AAU developed a transparent evaluation process for con- 
cept papers, and took care to ensure the project sites selected represented 
the diversity of the AAU membership in terms of institutional characteristics 
and the range of topics addressed by the projects. Second, before finalizing 
the selection of project sites, AAU announced the formation of a network 
and a network workshop to which all institutions were invited. 
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The AAU STEM Network is intended to be a collaborative network that helps 
to support and link AAU institutions grappling with similar challenges and bar- 
riers in reforming and improving STEM teaching and learning for undergrad- 
uate students. The network provides a forum to facilitate ongoing interaction 
and exchange of information and ideas, as well as to cultivate relationships 
among those leading reform efforts on different campuses. Network mem- 
bers share best practices and lessons learned, have access to resources, and can 
engage with those leading reform efforts focused on strengthening undergrad- 
uate STEM education. The network 
includes the primary point of contact The network provides a 
for each campus but also includes forum to facilitate ongoing 
others with various roles on cam- 


pus, including administrators, faculty imteraction and exchange of 
members, postdocs, and students. information and ideas, aS well 
As of 2017, 55 AAU member uni- as to Cultivate relationships 


versities have participated in the 


Initiative with more than 450 unique AMON those leading reform 


faculty members and institutional eafforts on different campuses. 
leaders. AAU has hosted three STEM 


network conferences and a fourth conference is planned for Fall, 2017. These 
conferences have given attendees opportunities to showcase their work and 
learn about the work of others, discuss common themes and challenges, and 
build relationships across campus roles and institutions. 


To support a greater number of AAU campuses, AAU will award two rounds 
of twelve institutional mini-grants for intra- or inter-institutional work aimed 
at improving undergraduate teaching of introductory STEM courses. The 
intent of these mini-grants is to build upon and advance prior institutional 
commitments and newly established efforts aimed at sustained institutional 
change in undergraduate STEM teaching. Universities selected will receive 
$10,000 annually for a two-year grant period totaling $20,000 to support 
intra- or inter-institutional coordination of undergraduate STEM education 
reform. In addition, AAU will host a workshop for all mini-grant recipients 
and fund a campus visit by the AAU STEM Initiative project team to help lever- 
age and highlight campus efforts. 


The first cohort of mini-grants was awarded in January 2017 and went to 
California Institute of Technology; Cornell University; lowa State University; 
Massachusetts Institute of Technology; McGill University; The University of 
Texas at Austin; University of California, Irvine; University of California, Los 
Angeles; The University of Kansas; University of Missouri, Columbia; University 
of Virginia; and Yale University. 


AAU has learned through its Initiative that grants awarded by AAU to mem- 
ber institutions can have powerful symbolic implications on a campus that can 
help it to facilitate change. By providing institutions with even modest funds, 
department-based efforts become an institutional priority, and individuals 
leading these efforts have a lever to obtain additional institutional resources. 
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AAU universities are engaged in multiple innovative efforts to improve under- 
graduate education to help all students succeed. Several similar efforts are 
underway across the AAU STEM Network. AAU has observed: 


Department- and college-wide innovations to undergraduate teaching are 
recognized as institutional priorities. 


Data infrastructures and analytics are being capitalized on to improve stu- 
dent learning. 


Teaching and learning centers are being redesigned to support departmen- 
tal reforms. 


Departments are hiring educational experts to provide leadership for sys- 
temic improvement to undergraduate STEM courses. 


Learning spaces are being developed and re-engineered to support evi- 
dence-based pedagogy. 


Universities are addressing the critical challenge of improving the evalua- 
tion of faculty teaching. 


The following STEM Network Highlights feature innovate work occurring 
across the network. Visit aau.edu/STEM to learn more about AAU universi- 
ties’ work in STEM education. 
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STEM NETWORK HIGHLIGHT 


Yale University Center tor 
Teaching and Learning 


Yale University established its Center for Teaching and Learning (CTL) 
in 2014 to support students, postdocs, and faculty across the campus 
to realize President Salovey’s goal “to be the research institution most 
committed to teaching and learning.” This new Center integrated various 
teaching, tutoring, writing and technology-enabled learning programs 
that were previously distributed across the university. The Center offers 
course design workshops and classroom observations, technology work- 
shops, seminars on diversity and inclusion, writing support, science and 
quantitative reasoning tutoring, and global online learning initiatives via 
Coursera. The CTL promotes evidence-based teaching methods for all 
university instructors, including faculty and graduate teaching fellows. 
The Center also supports student learning and provides opportunities for 
students and postdoctoral scholars throughout the University to develop 
as teachers, mentors and leaders. 


Staff members at the Center routinely assist members of the Yale com- 
munity with course design, section planning, effective use of technolo- 
gy, assignment development, mid-semester classroom observations and 
feedback, interpreting student evaluations, and addressing challenges in 
the classroom. 


The Center makes teaching and learning more public by promoting con- 
versation and collaboration among all the stakeholders, and by sharing se- 
lect university teaching initiatives with a global audience. Evidence-based 
decisions about teaching and learning are valued, and the Center helps 
faculty, departments, and programs design meaningful ways to measure 
impacts. 


The Center reports to the Deputy Provost for Teaching and Learning and 
is led by professional staff members. Center staff collaborate with many 
other Yale and external organizations to fulfill the unit’s mission. 
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STEM NETWORK HIGHLIGHT 


Lecturers witn the Potential tor 
Security of Enployment 


Lecturer with the Potential for Security of Employment (LPSOE) and Lec 
turer with Security of Employment (LSOE) are designations used by the 
University of California for faculty members appointed to help meet the 
long-term instructional needs of the university that cannot be best ful- 
filled by an appointee in the Professor (Ladder Rank) series. 


LPSOEs and LSOEs are expected to contribute excellent teaching, and play 
a leadership role in teaching in their departments and disciplines. They 
also have leadership responsibilities that transcend teaching, including as 
facilitators and initiators of instructional development, curriculum design, 
course structure, teaching methods, new technologies, and coordination 
of a spectrum of teaching activities. 


LPSOEs are equivalent in level to assistant professors. They have a proba- 
tionary period and may be appointed in the series for no more than eight 
years. They receive appraisals and are considered for promotion to LSOE 
on the same timelines as assistant professors. LSOEs are equivalent in 
level to associate professor, and “security of employment” is analogous to 
tenure. The title “Senior Lecturer” may be assigned to an appointee who 
provides services of exceptional value and whose excellent teaching and 
professional accomplishments have made him/her a recognized leader 
in his or her professional field and/or in education. The Senior LSOE is 
equivalent in level to full professor. 


Appointees in the LPSOE/LSOE series are reviewed for performance 

using four criteria: 

1. Teaching of truly exceptional quality, 

2. Professional achievement and activity: an appointee in the LSOE series 
is expected to maintain currency in the profession and pedagogy, 

3. University and public service, and 

4. Educational leadership beyond the campus and contributions to instruc 
tion related activities (i.e., conducting TA training, supervision of student 
affairs, development of instructional materials/multimedia). 
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University of Michigan Digital 
Innovation Greenhouse 


The University of Michigan’s Digital Innovation Greenhouse (DIG) ex- 
ists to design and develop innovative software tools that support the 
improvement of education through personalization at scale. “Personal- 
ization at scale” refers to taking innovations that personalize education, 
tailoring them to individual student needs and increasing engagement, 
and scaling these up so that all students can have access to them. Digital 
enterprises too often lack the resources to fully grow from early innova- 
tion to widespread adoption, and DIG exists to help efforts overcome 
the “valley of death.” 


STEM NETWORK HIGHLIGHT 


DIG works with user communities to grow tools to maturity, and es- 
tablishes a pathway to scale through collaboration across U-M's digital 
ecosystem. With a team (including faculty, staff, and student fellows) of 
developers, designers, behavioral scientists, and data scientists, DIG helps 
translate digital engagement tools from innovation to infrastructure. 


One of DIG’s main projects is developing Academic Reporting Tools 
(ART 2.0). ART 2.0 is a data visualization tool that will assist decision 
makers in accessing and analyzing course and academic program data 
to help administration, faculty, and students make more informed de- 
cisions. ART 2.0 will optimize instructional practices for faculty and 
answer questions such as “what majors are my students pursuing?” 
while also structuring the learning process for students through an in- 
teractive platform that answers questions such as “what courses have 
most of the students in this course already completed?” By allowing 
students and faculty alike to access data on courses and majors from 
past academic terms, ART 2.0 is fostering a community that allows for 
data driven information to lead towards better decision making. Other 
projects DIG is current engaged in include: 
= ECoach — a way to provide students with personalized assistance in 
large classes, which has already assisted over 15,000 students at U-M. 
=" GradeCraft — using gaming to enhance student motivation and learn- 
ing. 
= M-Write - writing-to-learn pedagogies that ask students them to ex- 
plain what they know, interact with one another through peer review, 
and learn through a revision process. 


DIG has also worked on learning tools focused on sustainability cas- 
es, interactive role-playing simulations, and a library of problems for 
self-testing, as well as developing an early warning system for advisors 
that flags when students are struggling in their courses. 
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STEM NETWORK HIGHLIGHT 


Cornell University Arts & Sciences 
Active Learning Initiative 


One way in which Cornell University has empowered departments to 
improve teaching and learning across dimensions of their undergraduate 
curriculum is through the Active Learning Initiative (ALI). ALI allows depart- 
ments to submit proposals and receive grant funding from the institution 
to “encourage and facilitate high-impact learning practices, technology 
enhanced learning, and a culture of educational excellence at the depart- 
mental and college levels.” 


ALI's initial efforts began five years ago with the conversion of four large 
course sequences in physics and biology, changes that have led to increased 
student learning gains, especially from students who had been receiving 
poor grades. Earlier in 2017, the university announced a $2.7 million ex- 
pansion of the initiative, funded by a gift from two alumni. 


One of the two donors, Alex Hanson, noted that donors and volunteers 
want to make a positive impact, and “Laura [the other donor] and | have 
been impressed with the game-changing quantitative and qualitative re- 
sults so far [of the ALI],” he said. “The College has demonstrated both a 
large treatment effect and a statistically significant improvement.” 


This second iteration involves six new projects in the Departments of Mu- 
sic, Classics, Economics, Mathematics, Physics, and Sociology. These new 
projects will impact thousands of students each year. 


For example, the mathematics project will change how courses are taught 
that reach students at critical transition points in their mathematical de- 
velopment. Goals for the project include increasing student confidence 
in their own mathematical abilities and improving student perception of 
mathematics as an inquiry-based discipline. The physics project will focus 
on rethinking how labs work to develop students’ scientific reasoning, 
critical thinking, and experimentation skills. 


According to Peter Lepage, Director of Education Innovation and Goldwin 
Smith Professor of Physics, “The projects chosen outline clear ideas that 
will improve student learning in ways that can be measured, and therefore 
propagated to other courses and departments. And they involve teams of 
faculty that will generate new energy and thought around pedagogy in 
the departments and the rest of the college.” 
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STEM NETWORK HIGHLIGHT 


University of Kansas Department 
Evaluation of Faculty Teaching Rubric 


The University of Kansas (KU) requires that evaluation of faculty teaching 
— including information from the instructor, students, and peers — be 
considered for promotion and tenure. However, traditionally the quality 
of this information has been highly variable and reviewers may struggle to 
make sense of it. In practice, many evaluations have prioritized a narrow 
dimension of teaching activity (the behavior of the instructor in the class- 
room) and a limited source of evidence (student evaluations). To address 
these shortcomings, the Center for Teaching Excellence at KU recently 
developed a rubric for department-level evaluation of faculty teaching. 


The goal of the rubric is to help departmental committees integrate 
information from the faculty member being evaluated, their peers, and 
their students to create a more holistic view of the faculty member's 
teaching contributions. The rubric identifies seven dimensions of teach- 
ing practice that address contributions to both individual courses and 
the department's curriculum: 

Goals, content, and alignment 

Teaching practices 

Achievement of learning outcomes 

Classroom climate and student perceptions 

Reflection and iterative growth 

Mentoring and advising, and 

Involvement in teaching service, scholarship, or community. 


For each of these categories, the rubric provides both guiding questions 
and defined expectations. The rubric can also be used to guide a con- 
structive peer-review process, reflection, and iterative improvement. To 
ensure applicability across disciplines, the rubric does not weigh or place 
focus on any particular element or require a particular type of evidence 
to be used. Departments are encouraged to modify the rubric and use it 
to build consensus about the dimensions, the questions and the criteria. 


Providing a rubric to structure the evaluation of faculty members’ teach- 
ing increases the visibility of all dimensions of teaching, clarifies faculty 
teaching expectations, enables quick identification of strengths and areas 
for improvement, and brings consistency across evaluations and over time. 


The implementation strategy for the rubric included discussions with de- 
partment chairs and KU Center for Teaching Excellence department ambas- 
sadors in advance of its release to increase the probability of broad buy-in. 
The rubric was piloted during the 2016-2017 academic year as a guide for 
peer review of teaching, promotion and tenure, and third-year reviews. 
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STEM NETWORK HIGHLIGHT 


Learning Analytics Fellows Program 


In 2015, Indiana University Bloomington (IUB) initiated a Learning Ana- 
lytics Fellows Program (Fellows), a multidisciplinary Community of Trans- 
formation made up of faculty members from across the Bloomington 
Campus. Faculty members engage in scholarly research that uses learning 
analytics to analyze student success in their courses, curriculum, and pro- 
grams to advance IUB's strategic plan and its commitment to improved 
student success. Participating faculty members gain the knowledge and 
skills necessary to use learning analytics to make data-driven decisions 
about student success at the course, program, and university levels. 


Since 2015, twenty-eight faculty members: 
participated from 11 different programs 
conducted 29 unique research projects 
studied 150,000 individual students 
analyzed 3.2 million student records 


Early adoption by Fellows to use analytical data, and their ongoing re- 
search about student success, will continue to contribute to institutional 
efforts to create a culture at |UB that values data-driven decision making, 
with the ultimate goal of improving success for all students. 
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section 4 | 


Undergraduate STEM Education Initiative 


At AAU Universities 


The following recommendations draw primarily from AAU’s cross-institutional 
portrait of successful undergraduate STEM educational reforms. In addition 
to the AAU Framework, several factors seemed to support — or when absent 
work against — adopting evidenced-based pedagogies in introductory STEM 
courses. The following key elements indicated movement toward systemic 
change in reforming undergraduate STEM education. 


Shift from individual to collective responsibility for curricula 
and instruction. 


A department and its faculty members must recognize and acknowledge 
their collective responsibility and ownership to improve the effectiveness 
of the department's introductory/foundational courses. 

Decisions about what fundamental concepts students need to learn in intro- 
ductory courses should no longer be left solely to the discretion of the indi- 
vidual faculty member teaching the course for a semester. Instead, these 
decisions must be made collectively by an entire department. 
Departments must actively promote the use of evidence-based active-learn- 
ing strategies in foundational courses and work jointly with faculty mem- 
bers to develop uniform ways to assess student learning in these courses. 


Hire educational experts within departments to bolster reforms. 


Educational experts should be formally embedded into the departments to 
support all faculty members in the adoption and use of evidence-based ped- 
agogy and proven effective instructional practices. 

Individuals in these roles should provide educational leadership to the depart- 
ment to help achieve desired reform through redesigning courses, co-teach- 
ing courses with other faculty members, taking on the responsibility for 
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departmental-wide educational improvement, and producing scholarship 
for the broader higher education community based on the evaluation of 
reformed courses. 

Embedded teaching experts within a department should be expected to 
engage in efforts with other STEM departments across the institution to bet- 
ter coordinate cross-disciplinary learning objectives, to share best practices, 
and to achieve broader institution-wide systemic STEM teaching reforms. 
Acceptance, support, and respect for the role of individuals with instructional 
expertise by departmental leadership and tenure-track faculty members is an 
essential element for long-lasting change. Given that institutional policies at 
research universities usually state that education is of equal value to research, 
faculty members and staff embedded within STEM departments with specific 
expertise in teaching should not be treated as ‘second class citizens.’ 


Reorganize institutional structures to support departmental 

reform efforts. 

= Symbolically and practically, establishing and maintaining organizations 
that support STEM educational reforms is an essential element of even- 
tual institutionalization. The work of various administrative units, such as 
Centers for Teaching and Learning, must be reoriented to better align with 
department based instructional improvement efforts and support the work 
of embedded teaching professionals within the departments. 

= At some institutions, additional within- and between-department struc 
tures should be created to support STEM educational reforms, particularly 
those that can help faculty members to evaluate, assess, and continually 
improve the effectiveness of their teaching. 


Harness institution-wide data to support student learning. 

= Research universities can greatly facilitate STEM education improvement 
by supporting the development and use of institution-wide data and ana- 
lytical tools on student instruction and learning outcomes. It is critical 
that data that is collected by the institution be compiled and shared with 
departments in ways that help them and their faculty members to contin- 
ually enhance the quality of their STEM instruction. 

= Universities should develop efficient tools to assist faculty members in their 
teaching and to assist administrators in monitoring student progress. 


Develop new institutional business and financial models to promote 
sustained improvement of undergraduate STEM education. 

a Undergraduate STEM education improvement at research universities can- 
not effectively be sustained based upon the same ‘from one grant to the 
next’ model that has been used to support research. Clearly an institutional 
commitment must be made. 

Funds will be required to hire the types of personnel best suited to enhance 
the quality of undergraduate STEM education while understanding these 
hires must fit into a research university setting. 

Additional learning spaces must be created to support active-learning 
techniques accompanied by support for faculty professional development 
around the effective use of the enhanced spaces. 
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Institutional leadership should find innovative ways to include improving 
the quality of undergraduate teaching in their major fundraising campaigns. 
They should also seek funds from industrial soonsors to endow “Chairs for 
Educational Excellence” within STEM departments. 


Better manage the simultaneous pursuit of high quality teaching 
and research. 

= The development and use of more effective ways to evaluate teaching in 
the faculty reward structures will be required to institutionalize STEM edu- 
cational reforms. 

Alternatives beyond the sole use of student evaluations/ratings must be 
developed and employed in evaluating the quality and effectiveness of 
faculty teaching. 

Institutions and departments need to modify their hiring practices to make 
clear that they have an expectation of excellence in teaching in addition 
to excellence in research. 


Leverage AAU to advance educational reforms and institutional change. 

# Situating the eight project sites within the larger AAU STEM Network was 
an effective way to encourage local reforms especially when peers from 
similar departments and types of institutions are doing similar activities on 
their campuses. 

= By AAU providing institutions with even modest funds, department-based 

efforts become an institutional priority, and individuals leading these efforts 

have a lever to obtain additional institutional resources. 

AAU involvement has symbolic implications that can help campuses achieve 

cultural and institutional change by providing legitimacy to STEM education 

reform efforts. 


At AAU 


In addition to AAU members and non-member research universities taking 
these steps, if long term systemic reform in undergraduate STEM education is 
to be achieved, it will also be important that AAU itself sustain the commit- 
ment it has made to supporting educational improvement at the undergrad- 
uate level. To help maintain the momentum for reform that has been gener- 
ated by the Initiative, AAU leadership has already committed to extend the 
initial five-year effort indefinitely by integrating continued support for under- 
graduate STEM education reform and improvement into its ongoing staffing 
structure and portfolio of work. Additionally, AAU will continue to explore 
how lessons learned to from the Initiative can be applied to improving under- 
graduate teaching in other non-STEM disciplines and how AAU, as an asso- 
ciation, can continue to promote excellence in undergraduate education. 
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In collaboration with Adrianna Kezar, Professor Rossier School of Education and 
Co-Director, Pullias Center for Higher Education at the University of Southern 
California, AAU examined how a national higher education association 
can in fact encourage and help create educational change at its mem- 
ber institutions. 


The National Science Foundation study, “Scaling Undergraduate STEM Education 
Reforms at AAU Institutions,” used the AAU Undergraduate STEM Education 
Initiative as a real-time, field-based innovation to examine the unique role AAU 
can play as compared to any other organization nationally to achieve scale of evi- 
dence based teaching practices. The overarching objective is to determine how the 
AAU STEM Initiative achieves scale of reforming undergraduate STEM teaching and 
learning. The study also examined the distributed leadership role of AAU and its 
influence as a result of being a highly-esteemed organization in higher education. 


The study involved document analysis, observations, and interviews. In total, over 
10,000 pages of documents were reviewed to understand the Initiative and obser- 
vations were conducted over two and half years. Four groups were interviewed: 
1) AAU STEM Initiative key leaders and personnel; 2) faculty members and admin- 
istrators at the AAU STEM Project Sites; 3) faculty members and administrators 
active in the AAU STEM Network; and 4) collaborators from outside organiza- 
tions that have worked with AAU on the Initiative. In total, the study included 104 
interviews. The following key themes emerged from the study: 


Assess your strengths and assets as an organization. One of the most import- 
ant lessons derived from this study is the importance of organizations engaged in 
change processes to start by assessing their strengths to help devise the best and 
most strategic approach. For AAU, these strengths included their ability: a) to influ- 
ence leaders such as president and provosts and other prestigious and influential 
organizations within the higher education sector; b) to create and leverage net- 
works; c) to define overarching logics or value systems for the enterprise; and d) to 
work across various key stakeholders of the higher education system. 


Use a systems approach to scale change. Every organization is able to work 
at different levels of a system. Some organizations are well-positioned to help 
with creating change at a specific level of a system, while some organizations can 
work across multiple parts of the system. Research has shown that more levels of 
the systems that are impacted, the more likely changes are to scale and be sus- 
tained. AAU was an organization that could work at multiple levels and utilized 
this capability. One of the beneficial outcomes of the AAU initiative was thinking 
about aligning different efforts within the overall landscape of improving under- 
graduate STEM education. Organizations should establish where they can work 
best within the overall system and strategically apply their efforts. 


Use multiple change theories for maximum impact. In projects that involve 
multiple stakeholders and complex motivations and issues, using multiple theories 
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of action to scale change can be extremely valuable. Leaders in AAU adopted a 
multi theory approach to the change process that benefited their trajectory for- 
ward and efficacy. It is all too common for change efforts to adopt a more simplistic 
approach to change and AAU’s deployment of various theories of action increased 
their chances of success. Embedding strategies which can be used in multi-faceted 
ways as AAU did is also a very efficient way to use time and resources, finite fac 
tors that are generally in low supply. 


Understand and consider influence strategies. Influence is generally an implicit 
strategy and not one that organizations conduct strategic planning around - even 
though it is an important lever for change. This study was able to provide concrete 
descriptions for what influence can look like within higher education setting. AAU’s 
most important change strategy was deploying its influence as a prestige organiza- 
tion to garner the attention of leaders and to motivate change. Every organization has 
the ability to influence some set up of groups of individuals and consideration of the 
most effective influence strategies is particularly important for a strategic approach. 


Carefully evaluate framing, messaging, and language used to communi- 
cate change. While there is a great deal of attention to language as it relates to 
visions articulated in change processes at the single institutional level, the process 
of articulating new “institutional logics” that guide more macro change processes 
have not been articulated in the higher education change literature. Just as indi- 
vidual institutions must carefully craft a vision around change for institutional stra- 
tegic planning processes, scaling efforts must also articulate a “common agenda” 
and even more importantly a compelling new set of logics to undergird institu- 
tional action across the sector. AAU successfully created a strong message that 
“AAU institutions need to be as excellent in teaching as they are in research,” which 
was a very compelling message to stakeholders. Organizations attempting to scale 
change should very carefully evaluate the logics they are developing and the lan- 
guage used articulate the new logic. 


Use networks to scale change. Organizations often organically allow networks 
to develop and may not intentionally plan how networks can be used and con- 
nected to scale change. AAU strategically considered and developed networks. 
Organizations can use the lessons from this study to help define and implement 
networks for scaling change. 


Create distributed leadership for improving STEM education. The litera- 
ture on scale suggests that leadership throughout the system is critical for scaling 
changes. Distributed leadership draws on leadership throughout the system and 
does not make distinctions between leaders in formal positions of authority versus 
informal leaders in terms of their value and importance for creating change. Leaders 
at different levels have insights into particular issues related to the change content 
as well as process. AAU partially played this role through its conveying power in 
that it brought together leadership from across the system. Reform efforts would 
benefit from an organization facilitating the development of leaders at these mul- 
tiple levels in support of improving undergraduate STEM education. 


Undergraduate STEM Education Initiative 


125 


126 


AAU Undergraduate STEM Education Initiative 
Advisory Committee Members 


Cynthia J. Atman, Director, Center for the Advancement of Engineering Education 
and Professor, Human Centered Design and Engineering, University of Washington 


Jim Borgford-Parnell, Assistant Director, Center for Engineering Teaching and 
Learning, University of Washington 


David M. Bressoud, DeWitt Wallace Professor of Mathematics, Macalester College 


Peter J. Bruns, Professor of Genetics Emeritus, Cornell University and Vice President 
for Grants and Special Programs (retired), Howard Hughes Medical Institute (HHMI) 


Linda Columbus, Associate Professor of Chemistry, Department of Chemistry, 
University of Virginia, Cottrell Scholar 


Edward J. Coyle, Arbutus Chair and Director of the Arbutus Center for the Integration 
of Research Education, Georgia Institute of Technology 


James S. Fairweather, Professor Emeritus, Higher, Adult and Lifelong Education, 
Michigan State University 


Noah Finkelstein, Professor of Physics Education Research, Department of Physics, 
University of Colorado at Boulder 


S. James Gates, Jr., Ford Foundation Professor of Physics, Brown University 


Jo Handelsman, Director, Institute for Discovery, University of Wisconsin-Madison, 
(former committee member) 


Sylvia Hurtado, Professor and Director of the Higher Education Research Institute 
(HERI), Graduate School of Education and Information Sciences, University of California, 
Los Angeles 


Kathy Mann Koepke, Director of Mathematics and Science Cognition and Learning, 
National Institutes of Health (NIH) 


G. Peter Lepage, Goldwin Smith Professor of Physics, Cornell University 


Haynes R. Miller, Professor of Mathematics and Associate Department Head, 
Department of Mathematics, Massachusetts Institute of Technology (MIT) 


Bassam Z. Shakhashiri, Professor of Chemistry, William T. Evjue Distinguished Chair 
for the Wisconsin Idea, University of Wisconsin-Madison, and 2012 President, American 
Chemical Society 


Linda L. Slakey, Senior Advisor, AAU Undergraduate STEM Education Initiative, 
and former Director, Division of Undergraduate Education, National Science 
Foundation (NSF) 


Candace Thille, Director of Open Learning Initiative (OLI); Assistant Professor of 
Education; and Senior Research Fellow, Office of the Vice Provost for Online Learning, 
Stanford University 


Association of American Universities 


Association 
oy ma alclalers 1p 
Universities 


Inquiry - Innovation - Impact 


Founded in 1900, the Association of American 
Ulalhiclasiidremeolaal olaiscmoPare| 1a) ace le lial-remlacieieele(elass 
that continually advance society through education, 
research, and discovery. 


Our universities earn the majority of competitively 
awarded federal funding for academic research, 
elccmianl econ diare male lnarelamiiicwlale mW ceil lel=iaremaalceleleia 
research, and are educating tomorrow's visionary 
leaders and global citizens. 


AAU members collectively help shape policy for 
higher education, science, and innovation; promote 
best practices in undergraduate and graduate 

=o [Uler-1u(o)ayare lace mcude=lalelaal-lamealem@elaiiasloluilelacmeyi 
research universities to society. 


———— 
——— 


Pa 
— 

= 
ae 

ee 


1200 New York Ave, NW, Washington, DC, 20005, Suite: 550 P: 202.408.7500 www.aau.edu ¥ & f @AAUniversities 


